Augmented reality interface for assisting a user to operate an ultrasound device

ABSTRACT

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(e) of each ofthe following U.S. Provisional Applications: U.S. ProvisionalApplication Ser. No. 62/352,382, titled “AUTOMATIC ACQUISITIONASSISTANCE AND REAL-TIME MEASUREMENT FOR ULTRASOUND IMAGING USING DEEPLEARNING” filed on Jun. 20, 2016, U.S. Provisional Application Ser. No.62/384,187, titled “METHOD AND APPARATUS TO PROVIDE AUGMENTED REALITYGUIDED ULTRASOUND DETECTION” filed on Sep. 6, 2016, U.S. ProvisionalApplication Ser. No. 62/384,144, titled “CLINICAL DIAGNOSTIC ANDTHERAPEUTIC DECISION SUPPORT USING PATIENT IMAGING DATA” filed on Sep.6, 2016, U.S. Provisional Application Ser. No. 62/434,980, titled“INTEGRATING STATISTICAL PRIOR KNOWLEDGE INTO CONVOLUTIONAL NEURALNETWORKS” filed on Dec. 15, 2016, U.S. Provisional Application Ser. No.62/445,195, titled “METHOD AND APPARATUS TO PROVIDE AUGMENTED REALITYGUIDED ULTRASOUND DETECTION” filed on Jan. 11, 2017, U.S. ProvisionalApplication Ser. No. 62/453,696, titled “METHOD AND APPARATUS TO PROVIDEAUGMENTED REALITY GUIDED ULTRASOUND DETECTION” filed on Feb. 2, 2017,and U.S. Provisional Application Ser. No. 62/463,094, titled “TECHNIQUESFOR LANDMARK LOCALIZATION” filed on Feb. 24, 2017. The disclosure ofeach and every identified application is incorporated herein byreference in its entirety.

FIELD

Generally, the aspects of the technology described herein relate toultrasound systems. Some aspects relate to techniques for guiding anoperator to use an ultrasound device.

BACKGROUND

Conventional ultrasound systems are large, complex, and expensivesystems that are typically used in large medical facilities (such as ahospital) and are operated by medical professionals that are experiencedwith these systems, such as ultrasound technicians. Ultrasoundtechnicians typically undergo years of hands-on training to learn how toproperly use the ultrasound imaging system. For example, an ultrasoundtechnician may learn how to appropriately position an ultrasound deviceon a subject to capture an ultrasound image in various anatomical views.Further, an ultrasound technician may learn how to read capturedultrasound images to infer medical information about the patient.

SUMMARY

Ultrasound examinations often include the acquisition of ultrasoundimages that contain a view of a particular anatomical structure (e.g.,an organ) of a subject. Acquisition of these ultrasound images typicallyrequires considerable skill. For example, an ultrasound technicianoperating an ultrasound device may need to know where the anatomicalstructure to be imaged is located on the subject and further how toproperly position the ultrasound device on the subject to capture amedically relevant ultrasound image of the anatomical structure. Holdingthe ultrasound device a few inches too high or too low on the subjectmay make the difference between capturing a medically relevantultrasound image and capturing a medically irrelevant ultrasound image.As a result, non-expert operators of an ultrasound device may haveconsiderable trouble capturing medically relevant ultrasound images of asubject. Common mistakes by these non-expert operators include:capturing ultrasound images of the incorrect anatomical structure andcapturing foreshortened (or truncated) ultrasound images of the correctanatomical structure.

Accordingly, the disclosure provides techniques to guide an operator ofan ultrasound device to capture medically relevant ultrasound images. Insome embodiments, these techniques may be embodied in a softwareapplication (hereinafter “App”) that may be installed on a computingdevice (e.g., a mobile smartphone, a tablet, a laptop, a smart watch,virtual reality (VR) headsets, augmented reality (AR) headsets, smartwearable devices, etc.). The App may provide real-time guidance to theoperator regarding how to properly position the ultrasound device on thesubject to capture a medically relevant ultrasound image. For example,the operator may place the ultrasound device on the subject and receivefeedback from the App regarding how to move the ultrasound device on thesubject. The feedback may be a sequence of instructions each including aparticular direction to move the ultrasound device (e.g., up, down,left, right, rotate clockwise, or rotate counter-clockwise). Thereby,the operator may follow these instructions to easily capture a medicallyrelevant ultrasound image.

In some embodiments, the App may leverage state-of-the-art machinelearning technology, such as deep learning. In these embodiments, theApp may employ a trained model, such as a trained neural network, thatis configured to generate instructions to provide to the operator. Inthis examples, the trained model may receive an ultrasound imagecaptured by the ultrasound device being used by the operator andprovide, as an output, an instruction to provide the operator. The modelmay be trained using a database of annotated ultrasound images. Theannotations for each of the ultrasound images may comprise, for example,an indication of whether the ultrasound image was a medically relevantultrasound image (e.g., an ultrasound image of a target anatomicalplane) or a medically irrelevant ultrasound image (e.g., an ultrasoundimage captured by an improperly positioned ultrasound device). If theultrasound image is medically irrelevant, the annotation may furtherinclude an indication of the error associated with the positioning ofthe ultrasound device that caused the captured ultrasound image to bemedically irrelevant (e.g., too high, too low, too clockwise, toocounter-clockwise, too far left, too far right). Thereby, the trainedmodel may recognize these medically irrelevant images and generate aninstruction regarding how the operator should reposition the ultrasounddevice to capture a medically relevant ultrasound image.

In some embodiments, an apparatus comprising a computing devicecomprising at least one processor is provided. The at least oneprocessor is configured to: obtain an ultrasound image of a subjectcaptured by an ultrasound device; determine, using an automated imageprocessing technique, whether the ultrasound image contains a targetanatomical view; responsive to a determination that the ultrasound imagedoes not contain the target anatomical view, provide at least oneinstruction to an operator of the ultrasound device indicating how toreposition the ultrasound device in furtherance of capturing anultrasound image of the subject that contains the target anatomicalview; and responsive to a determination that the ultrasound imagecontains the target anatomical view, provide an indication to theoperator that the ultrasound device is properly positioned.

In some embodiments, the apparatus further comprises a display coupledto the computing device and configured to display the at least oneinstruction to the operator. In some embodiments, the display isintegrated with the computing device.

In some embodiments, the computing device is configured to determinewhether the ultrasound image contains the target anatomical view atleast in part by analyzing the ultrasound image using a deep learningtechnique. In some embodiments, the computing device is configured todetermine whether the ultrasound image contains the target anatomicalview at least in part by providing the ultrasound image as an input to amulti-layer neural network. In some embodiments, the computing device isconfigured to determine whether the ultrasound image contains the targetanatomical view at least in part by using the multi-layer neural networkto obtain an output that is indicative of an anatomical view containedin the ultrasound image. In some embodiments, the computing device isconfigured to determine whether the ultrasound image contains the targetanatomical view at least in part by analyzing the ultrasound image usinga multi-layer neural network comprising at least one layer selected fromthe group consisting of: a pooling layer, a rectified linear units(ReLU) layer, a convolution layer, a dense layer, a pad layer, aconcatenate layer, and an upscale layer.

In some embodiments, the computing device is configured to determinewhether the ultrasound image contains the target anatomical view atleast in part by: identifying an anatomical view contained in theultrasound image using the automated image processing technique; anddetermining whether the anatomical view contained in the ultrasoundimage matches the target anatomical view. In some embodiments, thecomputing device is configured to, responsive to a determination thatthe anatomical view contained in the ultrasound image does not match thetarget anatomical view, generate the at least one instruction using theanatomical view contained in the ultrasound image.

In some embodiments, the computing device is configured to provide theat least one instruction at least in part by providing an instruction tomove the ultrasound device in a translational direction and/or arotational direction. In some embodiments, the computing device isconfigured to provide the at least one instruction to the operator atleast in part by providing the at least one instruction to the subject.

In some embodiments, a method is provided that comprises using at leastone computing device comprising at least one processor to perform:obtaining an ultrasound image of a subject captured by an ultrasounddevice; determining, using an automated image processing technique,whether the ultrasound image contains a target anatomical view;responsive to determining that the ultrasound image does not contain thetarget anatomical view, providing at least one instruction to anoperator of the ultrasound device indicating how to reposition theultrasound device in furtherance of capturing an ultrasound image of thesubject that contains the target anatomical view; and responsive todetermining that the ultrasound image contains the target anatomicalview, providing an indication to the operator that the ultrasound deviceis properly positioned.

In some embodiments, determining whether the ultrasound image containsthe target anatomical view comprises analyzing the ultrasound imageusing a deep learning technique. In some embodiments, determiningwhether the ultrasound image contains the target anatomical viewcomprises providing the ultrasound image as an input to a multi-layerneural network. In some embodiments, determining whether the ultrasoundimage contains the target anatomical view comprises using themulti-layer neural network to obtain an output that is indicative of ananatomical view contained in the ultrasound image. In some embodiments,determining whether the ultrasound image contains the target anatomicalview comprises analyzing the ultrasound image using a multi-layer neuralnetwork comprising at least one layer selected from the group consistingof: a pooling layer, a rectified linear units (ReLU) layer, aconvolution layer, a dense layer, a pad layer, a concatenate layer, andan upscale layer.

In some embodiments, determining whether the ultrasound image containsthe target anatomical view comprises: identifying an anatomical viewcontained in the ultrasound image using the automated image processingtechnique; and determining whether the anatomical view contained in theultrasound image matches the target anatomical view.

In some embodiments, the method further comprises, responsive todetermining that the anatomical view contained in the ultrasound imagedoes not match the target anatomical view, generating the at least oneinstruction using the anatomical view contained in the ultrasound image.

In some embodiments, providing the at least one instruction comprisesproviding an instruction to move the ultrasound device in atranslational direction and/or a rotational direction. In someembodiments, providing the at least one instruction to the operatorcomprises providing the at least one instruction to the subject.

In some embodiments, a system is provided that comprises an ultrasounddevice configured to capture an ultrasound image of a subject; and acomputing device communicatively coupled to the ultrasound device. Thecomputing device is configured to: obtain the ultrasound image of thesubject captured by the ultrasound device; determine, using an automatedimage processing technique, whether the ultrasound image contains atarget anatomical view; responsive to a determination that theultrasound image does not contain the target anatomical view, provide atleast one instruction to an operator of the ultrasound device indicatinghow to reposition the ultrasound device to capture an ultrasound imageof the subject that contains the target anatomical view; and responsiveto a determination that the ultrasound image contains the targetanatomical view, provide an indication to the operator that theultrasound device is properly positioned.

In some embodiments, the ultrasound device comprises a plurality ofultrasonic transducers. In some embodiments, the plurality of ultrasonictransducers comprises an ultrasonic transducer selected from the groupconsisting of: a capacitive micromachined ultrasonic transducer (CMUT),a CMOS ultrasonic transducer (CUT), and a piezoelectric micromachinedultrasonic transducer (PMUT).

In some embodiments, the computing device is a mobile smartphone or atablet. In some embodiments, the computing device is configured todetermine whether the ultrasound image contains the target anatomicalview at least in part by analyzing the ultrasound image using a deeplearning technique. In some embodiments, the computing device isconfigured to determine whether the ultrasound image contains the targetanatomical view at least in part by providing the ultrasound image as aninput to a multi-layer neural network. In some embodiments, thecomputing device is configured to determine whether the ultrasound imagecontains the target anatomical view at least in part by using themulti-layer convolutional neural network to obtain an output that isindicative of an anatomical view contained in the ultrasound image.

In some embodiments, the computing device is configured to determinewhether the ultrasound image contains the target anatomical at least inpart by: identifying an anatomical view contained in the ultrasoundimage using the automated image processing technique; and determiningwhether the anatomical view contained in the ultrasound image matchesthe target anatomical view. In some embodiments, the computing device isconfigured to generate the at least one instruction using the anatomicalview contained in the ultrasound image responsive to a determinationthat the anatomical view contained in the ultrasound image does notmatch the target anatomical view.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The processor-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: obtain an ultrasoundimage of a subject captured by an ultrasound device; determine, using anautomated image processing technique, whether the ultrasound imagecontains a target anatomical view; responsive to a determination thatthe ultrasound image does not contain the target anatomical view,provide at least one instruction to an operator of the ultrasound deviceindicating how to reposition the ultrasound device in furtherance ofcapturing an ultrasound image of the subject that contains the targetanatomical view; and responsive to a determination that the ultrasoundimage contains the target anatomical view, provide an indication to theoperator that the ultrasound device is properly positioned.

In some embodiments, an ultrasound guidance apparatus comprising atleast one processor is provided. The at least one processor isconfigured to guide capture of an ultrasound image containing a targetanatomical view of a subject based on analysis of another ultrasoundimage.

In some embodiments, the at least one processor is configured to guidecapture of the ultrasound image at least in part by generating aguidance plan for how to guide an operator of an ultrasound device tocapture the ultrasound image containing the target anatomical view. Insome embodiments, the at least one processor is configured to guidecapture of the ultrasound image at least in part by providing at leastone instruction to the operator based on the generated guidance plan. Insome embodiments, the apparatus further comprises a display coupled tothe at least one processor and configured to display the at least oneinstruction to the operator. In some embodiments, the display and the atleast one processor are integrated into a computing device. In someembodiments, the at least one processor is configured to guide captureof the ultrasound image at least in part by identifying an anatomicalview contained in the other ultrasound image using a deep learningtechnique. In some embodiments, the at least one processor is configuredto guide capture of the ultrasound image at least in part byidentifying, using the identified anatomical view, a direction in whichto move the ultrasound device. In some embodiments, the at least oneprocessor is configured to guide capture of the ultrasound image atleast in part by determining whether the other ultrasound image containsan anatomical view of the subject within a target region of the subject.In some embodiments, the at least one processor is configured to providethe at least one instruction to the operator at least in part byproviding an instruction to the operator to move the ultrasound devicetoward a position at which the ultrasound device can obtain images ofviews within the target region of the subject responsive to adetermination that the anatomical view contained in the other ultrasoundimage is outside the target region. In some embodiments, the at leastone processor is configured to provide the at least one instruction tothe operator at least in part by providing an instruction to theoperator to move the ultrasound device toward a position at which theultrasound device can obtain an image of the target anatomical viewresponsive to a determination that the anatomical view contained in theother ultrasound image is within the target region.

In some embodiments, a system comprising an ultrasound device configuredto capture an ultrasound image of a subject and at least one processoris provided. The at least one processor is configured to guide captureof another ultrasound image containing a target anatomical view of asubject based on analysis of the ultrasound image captured by theultrasound device.

In some embodiments, the ultrasound device comprises an ultrasonictransducer selected from the group consisting of: a capacitivemicromachined ultrasonic transducer (CMUT), a CMOS ultrasonic transducer(CUT), and a piezoelectric micromachined ultrasonic transducer (PMUT).In some embodiments, the at least one processor is integrated into amobile smartphone or a tablet.

In some embodiments, the at least one processor is configured to guidecapture at least in part by: determining whether the ultrasound imagecontains a target anatomical view; responsive to determining that theultrasound image does not contain the target anatomical view,generating, using the ultrasound image, a guidance plan for how to guidean operator of the ultrasound device to capture an ultrasound image ofthe subject containing the target anatomical view; and providing atleast one instruction to the operator based on the generated guidanceplan. In some embodiments, the guidance plan comprises a sequence ofinstructions to guide the operator of the ultrasound device to move theultrasound device to a target location. In some embodiments, eachinstruction in the sequence of instructions is an instruction to movethe ultrasound device in a translational or rotational direction. Insome embodiments, the at least one processor is configured to generatethe guidance plan at least in part by determining whether the ultrasoundimage contains an anatomical view of the subject within a target regionof the subject. In some embodiments, the at least one processor isconfigured to provide the at least one instruction to the operator atleast in part by providing an instruction to the operator to move theultrasound device toward a position at which the ultrasound device canobtain images of views within the target region of the subjectresponsive to a determination that the anatomical view contained in theultrasound image is not within the target region. In some embodiments,the at least one processor is configured to provide the at least oneinstruction to the operator at least in part by providing an instructionto the operator to move the ultrasound device toward a position at whichthe ultrasound device can obtain an image of the target anatomical viewresponsive to a determination that the anatomical view contained in theultrasound image is within the target region.

In some embodiments, a method is provided. The method comprises using atleast one computing device comprising at least one processor to perform:obtaining an ultrasound image of a subject captured by an ultrasounddevice; determining whether the ultrasound image contains a targetanatomical view; responsive to determining that the ultrasound imagedoes not contain the target anatomical view: generating, using theultrasound image, a guidance plan for how to guide an operator of theultrasound device to capture an ultrasound image of the subjectcontaining the target anatomical view; and providing at least oneinstruction to the operator based on the generated guidance plan.

In some embodiments, generating the guidance plan comprises identifyingan anatomical view contained in the ultrasound image using an automatedimage processing technique. In some embodiments, generating the guidanceplan comprises identifying, using the identified anatomical view, adirection in which to move the ultrasound device, and wherein providingthe at least one instruction to the operator comprises providing aninstruction to the operator to move the ultrasound device in theidentified direction. In some embodiments, identifying the direction inwhich to move the ultrasound device comprises identifying atranslational direction or a rotational direction in which to move theultrasound device.

In some embodiments, generating the guidance plan comprises determiningwhether the ultrasound image contains an anatomical view of the subjectwithin a target region of the subject. In some embodiments, determiningwhether the ultrasound image contains the anatomical view of the subjectwithin the target region of the subject comprises determining whetherthe ultrasound image contains an anatomical view of at least part of thesubject's torso. In some embodiments, the method further comprisesresponsive to a determination that the anatomical view contained in theultrasound image is not within the target region, providing the at leastone instruction to the operator at least in part by providing aninstruction to the operator to move the ultrasound device toward aposition at which the ultrasound device can obtain images of viewswithin the target region of the subject. In some embodiments, providingthe instruction to the operator to move the ultrasound device toward theposition comprises providing to the operator a visual indication ofwhere the target region is located. In some embodiments, the methodfurther comprises responsive to a determination that the anatomical viewcontained in the ultrasound image is within the target region, providingthe at least one instruction to the operator at least in part byproviding an instruction to the operator to move the ultrasound devicetoward a position at which the ultrasound device can obtain an image ofthe target anatomical view. In some embodiments, providing theinstruction to the operator to instruct the operator to move theultrasound device toward the position comprises providing to theoperator a visual indication of a direction in which to move theultrasound device.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The processor-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: obtain an ultrasoundimage of a subject captured by an ultrasound device; determine whetherthe ultrasound image contains a target anatomical view; responsive to adetermination that the ultrasound image does not contain the targetanatomical view, generate, using the ultrasound image, a guidance planfor how to guide an operator of the ultrasound device to capture anultrasound image of the subject containing the target anatomical view;and provide at least one instruction to the operator based on thegenerated guidance plan.

In some embodiments, an ultrasound guidance apparatus is provided thatcomprises at least one processor configured to: obtain an image of anultrasound device being used by an operator; and generate, using theobtained image of the ultrasound device, an augmented reality interfaceto guide the operator to capture an ultrasound image containing a targetanatomical view.

In some embodiments, the apparatus further comprises a display coupledto the at least one processor and configured to display the augmentedreality interface to the operator. In some embodiments, the display andthe at least one processor are integrated into a computing device.

In some embodiments, the at least one processor is configured togenerate the augmented reality interface at least in part by overlayingat least one instruction indicating how the operator is to repositionthe ultrasound device onto the image of the ultrasound device to form acomposite image. In some embodiments, the at least one processor isconfigured to generate the augmented reality interface at least in partby identifying a pose of the ultrasound device in the image of theultrasound device. In some embodiments, the at least one processor isconfigured to overlay the at least one instruction at least in part byoverlaying the at least one instruction onto the image of the ultrasounddevice using the pose of the ultrasound device. In some embodiments, theat least one instruction comprises an arrow indicating a direction inwhich the operator is to move the ultrasound device.

In some embodiments, the at least one processor is configured to obtainan ultrasound image captured by the ultrasound device. In someembodiments, the at least one processor is configured to generate theaugmented reality interface at least in part by identifying a locationof the ultrasound device in the image of the ultrasound device. In someembodiments, the at least one processor is configured to generate theaugmented reality interface at least in part by overlaying theultrasound image onto the image of the ultrasound device using thelocation of the ultrasound device.

In some embodiments, a method is provided that comprises obtaining animage of an ultrasound device being used by an operator, the image beingcaptured by an imaging device different from the ultrasound device;generating a composite image at least in part by overlaying, onto theimage of the ultrasound device, at least one instruction indicating howthe operator is to reposition the ultrasound device; and presenting thecomposite image to the operator.

In some embodiments, the method further comprises identifying a pose ofthe ultrasound device in the image of the ultrasound device. In someembodiments, the ultrasound device has a marker disposed thereon, andwherein obtaining the image of the ultrasound device comprises obtainingan image of the marker. In some embodiments, identifying the pose of theultrasound device comprises identifying a location of the marker in theimage of the ultrasound device.

In some embodiments, overlaying the at least one instruction onto theimage of the ultrasound device is performed using the pose of theultrasound device. In some embodiments, overlaying the at least oneinstruction onto the image of the ultrasound device comprises overlayingan arrow onto at least part of the ultrasound device in the image of theultrasound device, the arrow indicating a direction in which theoperator is to move the ultrasound device.

In some embodiments, the method further comprises obtaining anultrasound image captured by the ultrasound device. In some embodiments,generating the composite image comprises overlaying the ultrasound imagecaptured by the ultrasound device onto the image of the ultrasounddevice. In some embodiments, the method further comprises identifying alocation of the ultrasound device in the image of the ultrasound device.In some embodiments, overlaying the ultrasound image onto the image ofthe ultrasound device is performed using the location of the ultrasounddevice.

In some embodiments, a system is provided that comprises an imagingdevice different from an ultrasound device being used by an operator; adisplay; and at least one processor. The at least one processor isconfigured to: obtain an image of the ultrasound device being used bythe operator captured by the imaging device; generate a composite imageat least in part by overlaying, onto the image of the ultrasound device,at least one instruction indicating how the operator is to repositionthe ultrasound device; and cause the display to present the compositeimage to the operator.

In some embodiments, the system further comprises a mobile smartphone ortablet comprising the display and the at least one processor. In someembodiments, the imaging device comprises a camera. In some embodiments,the mobile smartphone or tablet comprises the camera.

In some embodiments, the at least one processor is configured toidentify a pose of the ultrasound device in the image of the ultrasounddevice. In some embodiments, the ultrasound device comprises a markerdisposed thereon, wherein the image of the ultrasound device comprisesan image of the marker, and wherein the at least one processor isconfigured to identify the pose of the ultrasound device at least inpart by identifying a location of the marker in the image of theultrasound device. In some embodiments, the marker is selected from thegroup consisting of: a holographic marker, a dispersive marker, and anArUco marker. In some embodiments, the at least one processor isconfigured to generate the composite image at least in part byoverlaying the at least one instruction onto the image of the ultrasounddevice using the pose of the ultrasound device.

In some embodiments, the system further comprises the ultrasound device.In some embodiments, the at least one processor is configured togenerate the composite image at least in part by overlaying theultrasound image captured by the ultrasound device onto the image of theultrasound device. In some embodiments, the at least one processor isconfigured to identify a location of the ultrasound device in the imageof the ultrasound device and wherein the at least one processor isconfigured to overlay the ultrasound image onto the image of theultrasound device using the location of the ultrasound device.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The processor-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: obtain an image of anultrasound device being used by an operator, the image being captured byan imaging device different from the ultrasound device; generate acomposite image at least in part by overlaying, onto the image of theultrasound device, at least one instruction indicating how the operatoris to reposition the ultrasound device; and cause the display to presentthe composite image to the operator.

In some embodiments, an apparatus comprising at least one processor isprovided. The at least one processor is configured to obtain anultrasound image of a subject captured by an ultrasound device anddetermine, using an automated image processing technique, whether theultrasound image contains a target anatomical view.

In some embodiments, the at least one processor is configured todetermine whether the ultrasound image contains the target anatomicalview at least in part by analyzing the ultrasound image using a deeplearning technique. In some embodiments, the at least one processor isconfigured to determine whether the ultrasound image contains the targetanatomical view at least in part by providing the ultrasound image as aninput to a multi-layer neural network. In some embodiments, the at leastone processor is configured to determine whether the ultrasound imagecontains the target anatomical view at least in part by using themulti-layer neural network to obtain an output that is indicative of ananatomical view contained in the ultrasound image. In some embodiments,the at least one processor is configured to determine whether theultrasound image contains the target anatomical view at least in part byanalyzing the ultrasound image using a multi-layer neural networkcomprising at least one layer selected from the group consisting of: apooling layer, a rectified linear units (ReLU) layer, a convolutionlayer, a dense layer, a pad layer, a concatenate layer, and an upscalelayer.

In some embodiments, the at least one processor is configured todetermine whether the ultrasound image contains the target anatomicalview at least in part by: identifying an anatomical view contained inthe ultrasound image using the automated image processing technique; anddetermining whether the anatomical view contained in the ultrasoundimage matches the target anatomical view. In some embodiments, the atleast one processor is configured to, responsive to a determination thatthe anatomical view contained in the ultrasound image does not match thetarget anatomical view, generate at least one instruction indicating howto reposition the ultrasound device in furtherance of capturing anultrasound image of the subject that contains the target anatomical viewusing the anatomical view contained in the ultrasound image.

In some embodiments, the at least one processor is configured to:provide at least one instruction to an operator of the ultrasound deviceindicating how to reposition the ultrasound device in furtherance ofcapturing an ultrasound image of the subject that contains the targetanatomical view responsive to a determination that the ultrasound imagedoes not contain the target anatomical view; and provide an indicationto the operator that the ultrasound device is properly positionedresponsive to a determination that the ultrasound image contains thetarget anatomical view. In some embodiments, the apparatus furthercomprises a display coupled to the at least one processor and configuredto display the at least one instruction to the operator. In someembodiments, the at least one processor is configured to provide the atleast one instruction at least in part by providing an instruction tomove the ultrasound device in a translational direction and/or arotational direction. In some embodiments, the at least one processor isconfigured to provide the at least one instruction to the operator atleast in part by providing the at least one instruction to the subject.

According to at least one aspect, a method is provided. The methodcomprises using at least one computing device comprising at least oneprocessor to perform: obtaining an ultrasound image of a subjectcaptured by an ultrasound device; determining, using an automated imageprocessing technique, whether the ultrasound image contains a targetanatomical view; responsive to determining that the ultrasound imagedoes not contain the target anatomical view, providing at least oneinstruction to an operator of the ultrasound device indicating how toreposition the ultrasound device in furtherance of capturing anultrasound image of the subject that contains the target anatomicalview; and responsive to determining that the ultrasound image containsthe target anatomical view, providing an indication to the operator thatthe ultrasound device is properly positioned.

In some embodiments, determining whether the ultrasound image containsthe target anatomical view comprises analyzing the ultrasound imageusing a deep learning technique. In some embodiments, determiningwhether the ultrasound image contains the target anatomical viewcomprises providing the ultrasound image as an input to a multi-layerneural network. In some embodiments, determining whether the ultrasoundimage contains the target anatomical view comprises using themulti-layer neural network to obtain an output that is indicative of ananatomical view contained in the ultrasound image. In some embodiments,determining whether the ultrasound image contains the target anatomicalview comprises analyzing the ultrasound image using a multi-layer neuralnetwork comprising at least one layer selected from the group consistingof: a pooling layer, a rectified linear units (ReLU) layer, aconvolution layer, a dense layer, a pad layer, a concatenate layer, andan upscale layer.

In some embodiments, determining whether the ultrasound image containsthe target anatomical view comprises: identifying an anatomical viewcontained in the ultrasound image using the automated image processingtechnique; and determining whether the anatomical view contained in theultrasound image matches the target anatomical view. In someembodiments, the method further comprises responsive to determining thatthe anatomical view contained in the ultrasound image does not match thetarget anatomical view, generating the at least one instruction usingthe anatomical view contained in the ultrasound image.

In some embodiments, providing the at least one instruction comprisesproviding an instruction to move the ultrasound device in atranslational direction and/or a rotational direction. In someembodiments, providing the at least one instruction to the operatorcomprises providing the at least one instruction to the subject.

In some embodiments, a system comprises an ultrasound device configuredto capture an ultrasound image of a subject; and a computing devicecommunicatively coupled to the ultrasound device is provided. Thecomputing device is configured to: obtain the ultrasound image of thesubject captured by the ultrasound device; determine, using an automatedimage processing technique, whether the ultrasound image contains atarget anatomical view; responsive to a determination that theultrasound image does not contain the target anatomical view, provide atleast one instruction to an operator of the ultrasound device indicatinghow to reposition the ultrasound device to capture an ultrasound imageof the subject that contains the target anatomical view; and responsiveto a determination that the ultrasound image contains the targetanatomical view, provide an indication to the operator that theultrasound device is properly positioned.

In some embodiments, the ultrasound device comprises a plurality ofultrasonic transducers. In some embodiments, the plurality of ultrasonictransducers comprises an ultrasonic transducer selected from the groupconsisting of: a capacitive micromachined ultrasonic transducer (CMUT),a CMOS ultrasonic transducer (CUT), and a piezoelectric micromachinedultrasonic transducer (PMUT).

In some embodiments, the computing device is a mobile smartphone or atablet. In some embodiments, the computing device is configured todetermine whether the ultrasound image contains the target anatomicalview at least in part by analyzing the ultrasound image using a deeplearning technique. In some embodiments, the computing device isconfigured to determine whether the ultrasound image contains the targetanatomical view at least in part by providing the ultrasound image as aninput to a multi-layer neural network. In some embodiments, thecomputing device is configured to determine whether the ultrasound imagecontains the target anatomical view at least in part by using themulti-layer convolutional neural network to obtain an output that isindicative of an anatomical view contained in the ultrasound image.

In some embodiments, the computing device is configured to determinewhether the ultrasound image contains the target anatomical at least inpart by: identifying an anatomical view contained in the ultrasoundimage using the automated image processing technique; and determiningwhether the anatomical view contained in the ultrasound image matchesthe target anatomical view. In some embodiments, the computing device isconfigured to generate the at least one instruction using the anatomicalview contained in the ultrasound image responsive to a determinationthat the anatomical view contained in the ultrasound image does notmatch the target anatomical view.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The processor-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: obtain an ultrasoundimage of a subject captured by an ultrasound device; determine, using anautomated image processing technique, whether the ultrasound imagecontains a target anatomical view; responsive to a determination thatthe ultrasound image does not contain the target anatomical view,provide at least one instruction to an operator of the ultrasound deviceindicating how to reposition the ultrasound device in furtherance ofcapturing an ultrasound image of the subject that contains the targetanatomical view; and responsive to a determination that the ultrasoundimage contains the target anatomical view, provide an indication to theoperator that the ultrasound device is properly positioned.

In some embodiments, an apparatus is provided comprising at least oneprocessor configured to: obtain an image of a marker on an ultrasounddevice being used by an operator; and generate an augmented realityinterface configured to guide the operator using a pose of theultrasound device identified based on the marker.

In some embodiments, the apparatus further comprises a display coupledto the at least one processor and configured to display the augmentedreality interface to the operator. In some embodiments, the display andthe at least one processor are integrated into a computing device. Insome embodiments, the at least one processor is configured to generatethe augmented reality interface at least in part by overlaying aninstruction to the operator of the ultrasound device onto the imageusing the pose of the ultrasound device. In some embodiments, the atleast one processor is configured to obtain an ultrasound image capturedby the ultrasound device and generate the instruction to the operatorusing the ultrasound image. In some embodiments, the at least oneprocessor is configured to identify the pose of the ultrasound device inthe image at least in part by identifying a location of the marker inthe image. In some embodiments, the at least one processor is configuredto identify the pose of the ultrasound device at least in part byanalyzing at least one characteristic of the marker in the image. Insome embodiments, the at least one processor is configured to analyzethe at least one characteristics of the marker in the image at least inpart by identifying a color of the marker in the image. In someembodiments, the at least one processor is configured to identify thepose of the ultrasound device at least in part by identifying anorientation of the ultrasound device in the image using the color of themarker in the image. In some embodiments, the marker comprises ahologram or a monochrome pattern.

In some embodiments, a method is provided that comprises using at leastone computing device comprising at least one processor to perform:obtaining an image of a marker on an ultrasound device being used by anoperator, the image being captured by an imaging device different froman ultrasound device; automatically identifying a pose of the ultrasounddevice at least in part by analyzing at least one characteristic of themarker in the image; and providing an instruction to the operator of theultrasound device using the identified pose of the ultrasound device.

In some embodiments, identifying the pose of the ultrasound devicecomprises identifying a location of the marker in the image. In someembodiments, identifying the pose of the ultrasound device comprisesidentifying a position of the ultrasound device in the image using theidentified location of the marker in the image.

In some embodiments, identifying the pose of the ultrasound devicecomprises identifying a color of the marker in the image. In someembodiments, identifying the pose of the ultrasound device comprisesidentifying an orientation of the ultrasound device in the image usingthe color of the marker.

In some embodiments, obtaining the image of the marker comprisesobtaining an image of a hologram or a monochrome pattern. In someembodiments, the method further comprises obtaining an ultrasound imagecaptured by the ultrasound device; and generating the instruction usingthe ultrasound image. In some embodiments, the method further comprisesoverlaying the ultrasound image onto the image using the identified poseof the ultrasound device.

In some embodiments, providing the instruction comprises determining alocation for the instruction to be overlaid onto the image using thepose of the ultrasound device.

In some embodiments, a system is provided that comprises an imagingdevice different from an ultrasound device being used by an operator;and at least one processor. The at least one processor is configured toobtain an image of a marker on the ultrasound device being used by theoperator captured by the imaging device; automatically identify a poseof the ultrasound device at least in part by analyzing at least onecharacteristic of the marker in the obtained image; and provide aninstruction to the operator of the ultrasound device using theidentified pose of the ultrasound device.

In some embodiments, the system further comprises a mobile smartphone ortablet comprising the imaging device and the at least one processor. Insome embodiments, the system further comprises the ultrasound devicehaving the marker disposed thereon. In some embodiments, the marker isselected from the group consisting of: a holographic marker, adispersive marker, and an ArUco marker.

In some embodiments, the system further comprises a display coupled tothe at least one processor. In some embodiments, the at least oneprocessor is configured to provide the instruction at least in part bycausing the display to provide the instruction to the operator.

In some embodiments, the at least one processor is configured toidentify the pose of the ultrasound device at least in part byidentifying a location of the marker in the image. In some embodiments,the at least one processor is configured to identify the pose of theultrasound device at least in part by identifying a position of theultrasound device in the captured image using the identified location ofthe marker in the image.

In some embodiments, the at least one processor is configured toidentify the pose of the ultrasound device at least in part byidentifying a color of the marker in the image. In some embodiments, theat least one processor is configured to identify the pose of theultrasound device at least in part by identifying an orientation of theultrasound device in the captured image using the color of the marker.

In some embodiments, the at least one processor is configured to obtainan ultrasound image captured by the ultrasound device and generate theinstruction using the ultrasound image.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The processor-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: obtain an image of amarker on an ultrasound device being used by an operator, the imagebeing captured by an imaging device different from an ultrasound device;automatically identify a pose of the ultrasound device at least in partby analyzing at least one characteristic of the marker in the obtainedimage; and provide an instruction to the operator of the ultrasounddevice using the identified pose of the ultrasound device.

In some embodiments, an apparatus is provided that comprises at leastone processor configured to: obtain an ultrasound image of a subject;and identify at least one medical parameter of the subject at least inpart by analyzing the ultrasound image using a deep learning technique.

In some embodiments, the at least one processor is configured toidentify the at least one medical parameter of the subject at least inpart by identifying at least one anatomical feature of the subject inthe ultrasound image using the deep learning technique. In someembodiments, the at least one processor is configured to identify the atleast one anatomical feature of the subject at least in part byproviding the ultrasound image as an input to a multi-layer neuralnetwork. In some embodiments, the at least one processor is configuredto identify the at least one anatomical feature of the subject at leastin part by using the multi-layer neural network to obtain an output thatis indicative of the at least one anatomical feature of the subject inthe ultrasound image. In some embodiments, the at least one processor isconfigured to identify the at least one anatomical feature of thesubject at least in part by analyzing the ultrasound image using amulti-layer neural network comprising at least one layer selected fromthe group consisting of: a pooling layer, a rectified linear units(ReLU) layer, a convolution layer, a dense layer, a pad layer, aconcatenate layer, and an upscale layer. In some embodiments, the atleast one anatomical feature comprises an anatomical feature selectedfrom the group consisting of: a heart ventricle, a heart valve, a heartseptum, a heart papillary muscle, a heart atrium, an aorta, and a lung.

In some embodiments, the at least one medical parameter comprises amedical parameter selected from the group consisting of: an ejectionfraction, a fractional shortening, a ventricle diameter, a ventriclevolume, an end-diastolic volume, an end-systolic volume, a cardiacoutput, stroke volume, an intraventricular septum thickness, a ventriclewall thickness, and a pulse rate. In some embodiments, the at least oneprocessor is configured to overlay the at least one medical parameteronto the ultrasound image of the subject to form a composite image. Insome embodiments, the apparatus further comprises a display coupled tothe at least one processor and configured to display the composite imageto the operator. In some embodiments, the display and the at least oneprocessor are integrated into a computing device.

In some embodiments, a method is provided that comprises using at leastone computing device comprising at least one processor to perform:obtaining an ultrasound image of a subject captured by an ultrasounddevice; identifying at least one anatomical feature of the subject inthe ultrasound image using an automated image processing technique; andidentifying at least one medical parameter of the subject using theidentified anatomical feature in the ultrasound image.

In some embodiments, identifying the at least one anatomical feature ofthe subject comprises analyzing the ultrasound image using a deeplearning technique. In some embodiments, identifying the at least oneanatomical feature of the subject comprises providing the ultrasoundimage as an input to a multi-layer neural network. In some embodiments,identifying the at least one anatomical feature of the subject comprisesusing the multi-layer neural network to obtain an output that isindicative of the at least one anatomical feature of the subject in theultrasound image. In some embodiments, identifying the at least oneanatomical feature of the subject comprises analyzing the ultrasoundimage using a multi-layer neural network comprising at least one layerselected from the group consisting of: a pooling layer, a rectifiedlinear units (ReLU) layer, a convolution layer, a dense layer, a padlayer, a concatenate layer, and an upscale layer.

In some embodiments, identifying the at least one anatomical featurecomprises identifying an anatomical feature selected from the groupconsisting of: a heart ventricle, a heart valve, a heart septum, a heartpapillary muscle, a heart atrium, an aorta, and a lung. In someembodiments, identifying the at least one medical parameter comprisesidentifying a medical parameter selected from the group consisting of:an ejection fraction, a fractional shortening, a ventricle diameter, aventricle volume, an end-diastolic volume, an end-systolic volume, acardiac output, a stroke volume, an intraventricular septum thickness, aventricle wall thickness, and a pulse rate. In some embodiments,obtaining the ultrasound image of the subject comprises obtaining aplurality of ultrasound images of the subject, and wherein identifyingthe at least one anatomical feature of the subject comprises identifyinga ventricle in each of at least some of the plurality of ultrasoundimages using a multi-layer neural network. In some embodiments,identifying the at least one medical parameter comprises: estimating aventricle diameter of the identified ventricles in each of the at leastsome of the plurality of images to obtain a plurality of ventriclediameters including a first ventricle diameter and a second ventriclediameter that is different from the first ventricle diameter; using thefirst ventricle diameter to estimate an end-diastolic volume; and usingthe second ventricle diameter to estimate an end-systolic volume. Insome embodiments, identifying the at least one medical parametercomprises identifying an ejection fraction of the subject using theestimated end-diastolic volume and the estimated end-systolic volume.

In some embodiments, the method further comprises overlaying the atleast one medical parameter onto the ultrasound image to form acomposite image; and presenting the composite image.

In some embodiments, obtaining the ultrasound image comprises guiding anoperator of the ultrasound device to capture the ultrasound image of thesubject. In some embodiments, guiding the operator of the ultrasounddevice comprises providing the ultrasound image as an input to a firstmulti-layer neural network and wherein identifying the at least oneanatomical feature of the subject comprises providing the ultrasoundimage as an input to a second multi-layer neural network that isdifferent from the first multi-layer neural network.

In some embodiments, a system is provided that comprises an ultrasounddevice configured to capture an ultrasound image of a subject; and acomputing device communicatively coupled to the ultrasound device. Thecomputing device is configured to: obtain the ultrasound image capturedby the ultrasound device; identify at least one anatomical feature ofthe subject in the ultrasound image using an automated image processingtechnique; and identify at least one medical parameter of the subjectusing the identified anatomical feature in the ultrasound image.

In some embodiments, the ultrasound device comprises a plurality ofultrasonic transducers. In some embodiments, the plurality of ultrasonictransducers comprises an ultrasonic transducer selected from the groupconsisting of: a capacitive micromachined ultrasonic transducer (CMUT),a CMOS ultrasonic transducer (CUT), and a piezoelectric micromachinedultrasonic transducer (PMUT).

In some embodiments, the computing device is a mobile smartphone or atablet. In some embodiments, the computing device comprises a display,and wherein the computing device is configured to display an indicationof the at least one medical parameter using the display.

In some embodiments, the ultrasound image contains an anatomical viewselected from the group consisting of: a parasternal long axis (PLAX)anatomical view, a parasternal short-axis (PSAX) anatomical view, anapical four-chamber (A4C) anatomical view, and apical long axis (ALAX)anatomical view.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The process-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: obtain an ultrasoundimage of a subject captured by an ultrasound device; identify at leastone anatomical feature of the subject in the ultrasound image using anautomated image processing technique; and identify at least one medicalparameter of the subject using the identified anatomical feature in theultrasound image.

In some embodiments, an apparatus is provided that comprises at leastone processor configured to: obtain an ultrasound image of a subject;and generate a diagnosis of a medical condition of the subject at leastin part by analyzing the ultrasound image using a deep learningtechnique.

In some embodiments, the at least one processor is configured togenerate the diagnosis at least in part by identifying at least onemedical parameter of the subject using the ultrasound image. In someembodiments, the at least one medical parameter of the subject comprisesa medical parameter selected from the group consisting of: an ejectionfraction, a fractional shortening, a ventricle diameter, a ventriclevolume, an end-diastolic volume, an end-systolic volume, a cardiacoutput, a stroke volume, an intraventricular septum thickness, aventricle wall thickness, and a pulse rate.

In some embodiments, the at least one processor is configured to obtainthe ultrasound image at least in part by guiding an operator of theultrasound device to obtain the ultrasound image. In some embodiments,the at least one processor is configured to guide the operator of theultrasound device at least in part by providing at least one instructionto the operator to reposition the ultrasound device. In someembodiments, the operator is the subject.

In some embodiments, the at least one processor is configured to receivemedical information about the subject and identify a target anatomicalview of the subject to image based on the received medical informationabout the subject. In some embodiments, the medical information aboutthe subject comprises at least one member selected from the groupconsisting of: a heart rate, a blood pressure, a body surface area, anage, a weight, a height, and a medication being taken by the subject. Insome embodiments, the at least one processor is configured to identifythe target anatomical view of the subject to be imaged at least in partby identifying an anatomical view of a heart of the subject as thetarget anatomical view responsive to the medical information about thesubject indicating that the subject has experienced paroxysmal nocturnaldyspnea.

In some embodiments, the at least one processor is configured to providea recommended treatment for the subject to the operator of theultrasound device using the diagnosed medical condition of the subject.

In some embodiments, a method is provided that comprises using at leastone computing device comprising at least one processor to perform:receiving medical information about a subject; identifying, based on thereceived medical information, a target anatomical view of the subject tobe imaged by an ultrasound device; obtaining an ultrasound imagecontaining the target anatomical view captured by the ultrasound device;and generating a diagnosis of a medical condition of the subject usingthe ultrasound image containing the target anatomical view.

In some embodiments, obtaining the ultrasound image containing thetarget anatomical view comprises guiding an operator of the ultrasounddevice to obtain the ultrasound image containing the target anatomicalview. In some embodiments, guiding the operator of the ultrasound deviceto obtain the ultrasound image containing the target anatomical viewcomprises providing at least one instruction to the operator toreposition the ultrasound device. In some embodiments, guiding theoperator comprises guiding the subject.

In some embodiments, receiving the medical information about the subjectcomprises receiving medical information selected from the groupconsisting of: a heart rate, a blood pressure, a body surface area, anage, a weight, a height, and a medication being taken by the subject. Insome embodiments, identifying the target anatomical view of the subjectto be imaged comprises identifying an anatomical view of a heart of thesubject as the target anatomical view responsive to the medicalinformation about the subject indicating that the subject hasexperienced paroxysmal nocturnal dyspnea. In some embodiments,diagnosing the medical condition of the subject comprises identifying anejection fraction of the subject using the ultrasound image containingthe target anatomical view responsive to the medical information aboutthe subject indicating that the subject has experienced paroxysmalnocturnal dyspnea.

In some embodiments, generating the diagnosis of a medical condition ofthe subject comprises identifying at least one medical parameter of thesubject using the ultrasound image containing the target anatomicalview. In some embodiments, identifying the at least one medicalparameter of the subject comprises identifying a medical parameterselected from the group consisting of: an ejection fraction, afractional shortening, a ventricle diameter, a ventricle volume, anend-diastolic volume, an end-systolic volume, a cardiac output, a strokevolume, an intraventricular septum thickness, a ventricle wallthickness, and a pulse rate.

In some embodiments, the method further comprises providing arecommended treatment for the subject to the operator of the ultrasounddevice using the diagnosed medical condition of the subject.

In some embodiments, the method further comprises reading a barcodedisposed on the ultrasound device; and sending the barcode to anotherdevice to cause the other device to transmit the medical informationabout the subject to the at least one computing device. In someembodiments, the method further comprises sending the ultrasound imagecontaining the target anatomical view to the other device to cause theother device to add the ultrasound image containing the targetanatomical view to a medical file associated with the subject.

In some embodiments, a system is provided that comprises an ultrasounddevice configured to capture ultrasound images; and a computing devicecommunicatively coupled to the ultrasound device. The computing deviceis configured to: receive medical information about a subject; identify,based on the received medical information, a target anatomical view ofthe subject to be imaged by the ultrasound device; obtain an ultrasoundimage containing the target anatomical view captured by the ultrasounddevice; and generate a diagnosis of a medical condition of the subjectusing the ultrasound image containing the target anatomical view.

In some embodiments, the ultrasound device comprises a plurality ofultrasonic transducers. In some embodiments, the plurality of ultrasonictransducers comprises an ultrasonic transducer selected from the groupconsisting of: a capacitive micromachined ultrasonic transducer (CMUT),a CMOS ultrasonic transducer (CUT), and a piezoelectric micromachinedultrasonic transducer (PMUT).

In some embodiments, the computing device is a mobile smartphone or atablet. In some embodiments, the computing device is configured toidentify the target anatomical view at least in part by identifying ananatomical view of a heart of the subject as the target anatomical viewresponsive to the medical information about the subject indicating thatthe subject has experienced paroxysmal nocturnal dyspnea. In someembodiments, the computing device is configured to identify an ejectionfraction of the subject using the ultrasound image containing the targetanatomical view responsive to the medical information about the subjectindicating that the subject has experienced paroxysmal nocturnaldyspnea.

In some embodiments, the computing device is configured to generate thediagnosis of the medical condition of the subject at least in part byidentifying at least one medical parameter of the subject using theultrasound image containing the target anatomical view.

In some embodiments, at least one non-transitory computer-readablestorage medium storing processor-executable instructions is provided.The processor-executable instructions, when executed by at least oneprocessor, cause the at least one processor to: receive medicalinformation about a subject; identify, based on the received medicalinformation, a target anatomical view of the subject to be imaged by theultrasound device; obtain an ultrasound image containing the targetanatomical view captured by the ultrasound device; and generate adiagnosis of a medical condition of the subject using the ultrasoundimage containing the target anatomical view.

In some embodiments, a method for assessing position and orientation ofan ultrasonic probe is provided. The method comprises (a) receiving, bya host device, ultrasound image data generated by the ultrasound probepositioned to image a desired feature of a subject, wherein the hostdevice comprises a processor and memory; and (b) providing instructionsto reposition the ultrasound probe in order to capture the desiredfeature, wherein the instructions are determined based at least on thedesired feature.

In some embodiments, a method for real-time measurement prediction ofultrasound imaging is provided. The method comprises (a) receiving, by ahost device, ultrasound image data generated by a ultrasound probepositioned to image a desired feature of a subject, wherein the hostdevice comprises a processor and memory; and (b) comparing the receivedultrasound image data with trained model data to predict, in real time,a landmark of the received ultrasound image data.

In some embodiments, a method to provide real-time ultrasound imageacquisition assistance is provided. The method comprises (a) receivingan initial ultrasound image of a patient; (b) comparing attributes ofthe initial ultrasound image with criteria for a high quality ultrasoundimage; and (c) directing movement of an ultrasound probe to obtain asubsequent ultrasound image compliant with the criteria for the highquality ultrasound image.

In some embodiments, a method to provide real-time ultrasound imageacquisition assistance is provided. The method comprises (a) receivingan acquisition intent instruction for a final ultrasound imagery; (b)receiving a first ultrasound image from an ultrasound probe, the firstultrasound image comprising a perspective of a subject; (c) identifyingan infirmity of the first ultrasound image by comparing the firstultrasound image with the acquisition intent instruction; (d)identifying a remedial action to manipulate the ultrasound probe toremedy the infirmity of the first ultrasound imagery based on theacquisition intent instruction; and (e) displaying the identifiedremedial action to assist in acquisition of the final ultrasound image.

In some embodiments, a clinical diagnostic and therapeutic decisionsupport system is provided. The system comprises a processor, theprocessor configured to: (a) acquire medical ultrasound image data of asubject to be diagnosed; (b) display a diagnosis determined based on atleast the medical ultrasound image data; and (c) display a recommendedtreatment for the subject based on the diagnosis.

In some embodiments, a method of providing a clinical diagnostic andtherapeutic decision is provided. The method comprises (a) acquiring,with a processor, medical ultrasound image data of a subject to bediagnosed; (b) displaying a diagnosis determined based on at least themedical ultrasound image data; and (c) displaying a recommendedtreatment for the subject, wherein the recommended treatment isdetermined based on the diagnosis.

In some embodiments, a method for training a convolutional neuralnetwork using statistical prior knowledge. The method comprises (a)receiving a training set comprising a plurality of medical images of aplurality of subjects and a training annotation associated with each ofthe plurality of medical images; (b) receiving statistical priorknowledge of the plurality of the medical images, wherein thestatistical prior knowledge comprises statistics associated withvariability of the medical images arising from naturally occurringstructures of the plurality of subjects; and (c) training theconvolutional neural network, using the training set, by incorporatingthe statistical prior knowledge.

In some embodiments, a method for performing segmentation of a medicalimage. The method comprises (a) providing the medical image of a featureof a subject; and (b) using a trained convolutional neural network toperform the image segmentation of the medical image, wherein the trainedconvolutional neural network is trained using statistical priorknowledge.

In some embodiments, a method for performing landmark localization of amedical image. The method comprises (a) providing the medical image of afeature of a subject; and (b) using a trained convolutional neuralnetwork to perform the landmark localization of the medical image,wherein the trained convolutional neural network is trained usingstatistical prior knowledge.

In some embodiments, a method is provided comprising (a) capturing animage of an ultrasound probe positioned relative to a patient; (b)capturing an in vivo ultrasound image of a portion of the patient'sbody; (c) identifying a location of the ultrasound probe in the capturedimage of the ultrasound probe, the location identified relative to thepatient's body; (d) forming a composite image by overlaying the in vivoultrasound image onto the image of the ultrasound probe to form acomposite image at least in part by positioning the in vivo ultrasoundimage adjacent to the image of the ultrasound probe; and (e) displayingthe composite image.

In some embodiments, the method further comprises displaying a pluralityof composite images in real time. In some embodiments, the compositeimages are displayed on an augmented reality display. In someembodiments, the method further comprises providing instructions in realtime based on the plurality of composite images, wherein theinstructions guide a user of the ultrasound probe in acquisition ofsubsequent ultrasound images of the portion of the patient's body.

In some embodiments, a method is provided comprising (a) capturing animage of an ultrasound probe positioned relative to a patient; (b)capturing an in vivo ultrasound image of a portion of the patient'sbody; (c) identifying a location of the ultrasound probe in the capturedimage of the ultrasound probe, the location identified relative to thepatient's body; (d) forming a composite image by overlaying the in vivoultrasound image onto the image of the ultrasound probe to form acomposite image at least in part by positioning the in vivo ultrasoundimage adjacent to the image of the ultrasound probe; and (e) displayingthe composite image.

In some embodiments, a method of consumer-based use of an ultrasounddevice is provided. The method comprises (a) operating a portableultrasound device by a user; (b) capturing an image of the portableultrasound device using an image capture device; (c) adjusting aposition and/or orientation of the ultrasound device in response tofeedback provided by a processing device, wherein the feedback isgenerated by the processing device based at least on analysis ofultrasound data captured by the user using the portable ultrasounddevice; and (d) storing the ultrasound data captured by the user usingthe portable ultrasound device.

In some embodiments, operating the portable ultrasound device isperformed in the user's home. In some embodiments, the feedback isprovided in real time. In some embodiments, the feedback is providedusing augmented reality. In some embodiments, the feedback comprisesinstructions regarding one or more of: (i) where the user should placethe portable ultrasound device, (ii) how the user should reposition ororient the portable ultrasound device, (iii) how the user shouldlinearly translate the portable ultrasound device, and (iv) how the usershould act to facilitate capture of the ultrasound data. In someembodiments, the method further comprises displaying a diagnosisdetermined based on at least the medical image data. In someembodiments, the method further comprises displaying a recommendedtreatment for the subject, wherein the recommended treatment isdetermined based on the diagnosis.

In some embodiments, a method for training a convolutional neuralnetwork for landmark localization is provided. The method comprises (a)receiving a training set comprising a plurality of medical images of aplurality of subjects and a training annotation associated with each ofthe plurality of medical images; and (b) training the convolutionalneural network to regress one or more landmark locations based at leaston the training set.

In some embodiments, a method for performing landmark localization of amedical image of a subject is provided. The method comprises (a)providing the medical image of the subject; and (b) using a trainedconvolutional neural network to perform the landmark localization of themedical image of the subject, wherein the trained convolutional neuralnetwork is trained using a training set comprising a plurality ofmedical images of a plurality of subjects and a training annotationassociated with each of the plurality of medical images.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects and embodiments will be described with reference to thefollowing exemplary and non-limiting figures. It should be appreciatedthat the figures are not necessarily drawn to scale. Items appearing inmultiple figures are indicated by the same or a similar reference numberin all the figures in which they appear.

FIG. 1 shows an exemplary ultrasound system according to someembodiments of the disclosure;

FIG. 2 shows an exemplary guide path along which to move the ultrasounddevice from an initial position on the subject to a target position onthe subject according to some embodiments of the disclosure;

FIG. 3A shows an exemplary coarse instruction to be provided to anoperator according to some embodiments of the disclosure;

FIG. 3B shows an exemplary fine instruction to be provided to anoperator according to some embodiments of the disclosure;

FIG. 3C shows an exemplary confirmation to be provided to an operatoraccording to some embodiments of the disclosure;

FIG. 4 shows exemplary medical parameters overlaid onto an ultrasoundimage according to some embodiments of the disclosure;

FIGS. 5A and 5B show an exemplary ultrasound system configured toprovide an augmented reality interface to an operator according to someembodiments of the disclosure;

FIG. 6 shows an exemplary augmented reality interface according to someembodiments of the disclosure;

FIGS. 7A-7H show an exemplary user interface for a diagnosticapplication according to some embodiments of the disclosure;

FIGS. 8A-8D show an exemplary user interface for an at-home diagnosticapplication according to some embodiments of the disclosure;

FIG. 9 shows an exemplary method of guiding an operator of an ultrasounddevice to capture an ultrasound image containing a target anatomicalview according to some embodiments of the disclosure;

FIG. 10 shows an exemplary method of providing an augmented realityinterface to an operator of an ultrasound device embodiments of thedisclosure;

FIG. 11 shows an exemplary method of tracking a location of anultrasound device according to some embodiments of the disclosure;

FIG. 12 shows an exemplary method of identifying a medical parameter ofa subject using an ultrasound image according to some embodiments of thedisclosure;

FIG. 13 shows an exemplary method of generating a diagnosis of a medicalcondition of a subject according to some embodiments of the disclosure;

FIG. 14 shows an exemplary convolutional neural network according tosome embodiments of the disclosure;

FIG. 15A shows a block diagram of an exemplary ultrasound systemaccording to some embodiments of the disclosure;

FIG. 15B shows a block diagram of another exemplary ultrasound systemaccording to some embodiments of the disclosure;

FIG. 16 shows a block diagram of an exemplary ultrasound deviceaccording to some embodiments of the disclosure;

FIG. 17 shows a detailed block diagram of the exemplary ultrasounddevice shown in FIG. 16 according to some embodiments of the disclosure;

FIGS. 18A-18B show an exemplary handheld device comprising an ultrasounddevice and a display according to some embodiments of the disclosure;

FIGS. 18C-18E show an exemplary patch comprising an ultrasound deviceaccording to some embodiments of the disclosure; and

FIG. 18F shows an exemplary handheld device comprising an ultrasounddevice according to some embodiments of the disclosure.

DETAILED DESCRIPTION

Conventional ultrasound systems are large, complex, and expensivesystems that are typically only purchased by large medical facilitieswith significant financial resources. Recently, cheaper and less complexultrasound imaging devices have been introduced. Such imaging devicesmay include ultrasonic transducers monolithically integrated onto asingle semiconductor die to form a monolithic ultrasound device. Aspectsof such ultrasound-on-a chip devices are described in U.S. patentapplication Ser. No. 15/415,434 titled “UNIVERSAL ULTRASOUND DEVICE ANDRELATED APPARATUS AND METHODS,” filed on Jan. 25, 2017 (and assigned tothe assignee of the instant application), which is incorporated byreference herein in its entirety. The reduced cost and increasedportability of these new ultrasound devices may make them significantlymore accessible to the general public than conventional ultrasounddevices.

The inventors have recognized and appreciated that although the reducedcost and increased portability of ultrasound imaging devices makes themmore accessible to the general populace, people who could make use ofsuch devices have little to no training for how to use them. Forexample, a small clinic without a trained ultrasound technician on staffmay purchase an ultrasound device to help diagnose patients. In thisexample, a nurse at the small clinic may be familiar with ultrasoundtechnology and human physiology, but may know neither which anatomicalviews of a patient need to be imaged in order to identifymedically-relevant information about the patient nor how to obtain suchanatomical views using the ultrasound device. In another example, anultrasound device may be issued to a patient by a physician for at-homeuse to monitor the patient's heart. In all likelihood, the patientunderstands neither human physiology nor how to image his or her ownheart with the ultrasound device.

Accordingly, the inventors have developed assistive ultrasound imagingtechnology for guiding an operator of an ultrasound device to properlyuse the ultrasound device. This technology enables operators, havinglittle or no experience operating ultrasound devices, to capturemedically relevant ultrasound images and may further assist theoperators in interpreting the contents of the obtained images. Forexample, some of the techniques disclosed herein may be used to: (1)identify a particular anatomical view of a subject to image with anultrasound device; (2) guide an operator of the ultrasound device tocapture an ultrasound image of the subject that contains the particularanatomical view; and (3) analyze the captured ultrasound image toidentify medical information about the subject.

It should be appreciated that the embodiments described herein may beimplemented in any of numerous ways. Examples of specificimplementations are provided below for illustrative purposes only. Itshould be appreciated that these embodiments and thefeatures/capabilities provided may be used individually, all together,or in any combination of two or more, as aspects of the technologydescribed herein are not limited in this respect.

A. Instructing an Operator of an Ultrasound Device how to Position theDevice

The disclosure provides techniques for instructing an operator of anultrasound device how to position the ultrasound device on a subject tocapture a medically relevant ultrasound image. Capturing an ultrasoundimage of a subject that contains a particular anatomical view may bechallenging for novice ultrasound device operators. The operator (e.g.,a nurse, a technician or a lay person) needs to know not only where toinitially position the ultrasound device on the subject (e.g., apatient), but also how to adjust the position of the device on thesubject to capture an ultrasound image containing the target anatomicalview. In cases where the subject is also the operator, it may be evenmore challenging for the operator to identify the appropriate view asthe operator may not have a clear view of the ultrasound device.Accordingly, certain disclosed embodiments relate to new techniques forguiding the operator to capture an ultrasound image that contains thetarget anatomical view. The guidance may be provided via a softwareapplication (hereinafter “App”) installed on a computing device of theoperator (such as: a mobile device, a smartphone or smart-device,tablet, etc.). For example, the operator may install the App on acomputing device and connect the computing device to an ultrasounddevice (e.g., using a wireless connection such as BLUETOOTH or a wiredconnection such as a Lightning cable). The operator may then positionthe ultrasound device on the subject and the software application (viathe computing device) may provide feedback to the operator indicatingwhether the operator should reposition the ultrasound device and howhe/she should proceed to do so. Following the instructions allows anovice operator to capture medically-relevant ultrasound imagescontaining the target anatomical view.

In some embodiments, the instructions provided to the operator may begenerated at least in part by using state-of-the-art image processingtechnology such as deep learning. For example, the computing device mayanalyze a captured ultrasound image using deep learning techniques todetermine whether the ultrasound image contains the target anatomicalview. If the ultrasound image contains the target anatomical view, thecomputing device may provide a confirmation to the operator that theultrasound device is properly positioned on the subject and/oratomically start recording ultrasound images. Otherwise, the computingdevice may instruct the operator how to reposition the ultrasound device(e.g., “MOVE UP,” “MOVE LEFT,” “MOVE RIGHT,” “ROTATE CLOCKWISE,” “ROTATECOUNTER-CLOCKWISE,” or “MOVE DOWN”) to capture an ultrasound image thatcontains the target anatomical view.

The deep learning techniques described herein may be implemented inhardware, software or a combination of hardware and software. In oneembodiment, a deep learning technique is implemented in an Appexecutable on a smart device accessible to the operator. The App may,for example, leverage a display integrated into the smart device todisplay a user interface screen to the operator. In another embodiment,the App may be executed on a cloud and communicated to the operatorthrough the smart device. In yet another embodiment, the App may beexecuted on the ultrasound device itself and the instructions may becommunicated to the user either through the ultrasound device itself ora smart device associated with the ultrasound device. Thus, it should benoted that the execution of the App may be at a local or a remote devicewithout departing from the disclosed principles.

In some embodiments, techniques for providing instructions to anoperator of an ultrasound device regarding how to reposition theultrasound device to capture an ultrasound image containing a targetanatomical view of a subject may be embodied as a method that isperformed by, for example, a computing device that is communicativelycoupled to an ultrasound device. The computing device may be a mobilesmartphone, a tablet, a laptop, a workstation, or any other suitablecomputing device. The ultrasound device may be configured to transmitacoustic waves into a subject using ultrasonic transducers, detect thereflected acoustic waves, and use them to generate ultrasound data.Example ultrasonic transducers include capacitive micromachinedultrasonic transducers (CMUTs), CMOS ultrasonic transducers (CUTs), andpiezoelectric micromachined ultrasonic transducers (PMUTs). Theultrasonic transducers may be monolithically integrated with asemiconductor substrate of the ultrasound device. The ultrasound devicemay be implemented as, for example, a handheld device or as a patch thatis configured to adhere to the subject.

In some embodiments, an exemplary method may include obtaining anultrasound image of a subject captured using the ultrasound device. Forexample, the ultrasound device may generate ultrasound sound data andtransmit (via a wired or wireless communication link) the ultrasounddata to the computing device. The computing device may, in turn,generate the ultrasound image using the received ultrasound data. Themethod may further include determining whether the ultrasound imagecontains a target anatomical view using an automated image processingtechnique. For example, the ultrasound image may be analyzed using theautomated image processing technique to identify the anatomical viewcontained in the ultrasound image. The identified anatomical view may becompared with the target anatomical view to determine whether theidentified anatomical view matches the target anatomical view. If theidentified anatomical view matches the target anatomical view, then adetermination is made that the ultrasound image does contain the targetanatomical view. Otherwise, a determination is made that the ultrasoundimage does not contain the target anatomical view.

It should be appreciated that any of a variety of automated imageprocessing techniques may be employed to determine whether an ultrasoundimage contains the target anatomical view. Example automated imageprocessing techniques include machine learning techniques such as deeplearning techniques. In some embodiments, a convolutional neural networkmay be employed to determine whether an ultrasound image contains thetarget anatomical view. For example, the convolutional neural networkmay be trained with a set of ultrasound images labeled with theparticular anatomical view depicted in the ultrasound image. In thisexample, an ultrasound image may be provided as an input to the trainedconvolutional neural network and an indication of the particularanatomical view contained in the input ultrasound image may be providedas an output.

In another example, the convolutional neural network may be trained witha set of ultrasound images labeled with either one or more instructionsregarding how to move the ultrasound device to capture an ultrasoundimage containing the target anatomical view or an indication that theultrasound image contains the target anatomical view. In this example,an ultrasound image may be provided as an input to a trainedconvolutional neural network and an indication that the ultrasound imagecontains the target anatomical view or an instruction to provide theoperator may be provided as an output. The convolutional neural networkmay be implemented using a plurality of layers in any suitablecombination. Example layers that may be employed in the convolutionalneural network include: pooling layers, rectified linear units (ReLU)layers, convolutional layers, dense layers, pad layers, concatenatelayers, and/or upscale layers. Examples of specific neural networkarchitectures are provided herein in the Example Deep LearningTechniques section.

In some embodiments, the method may further include providing at leastone instruction (or one set of instructions) to an operator of theultrasound device indicating how to reposition the ultrasound device infurtherance of capturing an ultrasound image of the subject thatcontains the target anatomical view when a determination is made thatthe ultrasound image does not contain the target anatomical view. Theinstruction may be provided to the operator in any of a variety of ways.For example, the instruction may be displayed to the operator using adisplay (e.g., a display integrated into the computing device, such asthe operator's mobile device) or audibly provided to the operator usinga speaker (e.g., a speaker integrated into the computing device).Example instructions include “TURN CLOCKWISE,” “TURN COUNTER-CLOCKWISE,”“MOVE UP,” “MOVE DOWN,” “MOVE LEFT,” and “MOVE RIGHT.”

In some embodiments, the method may further include providing anindication to the operator that the ultrasound device is properlypositioned when a determination is made that the ultrasound imagecontains the target anatomical view. The indication to the operator thatthe ultrasound device is properly positioned may take any of a varietyof forms. For example, a symbol may be displayed to the operator such asa checkmark. Alternatively (or additionally), a message may be displayedand/or audibly played to the operator such as “POSITIONING COMPLETE.”

The instructions can be computed based on the current position of theultrasound device with respect to the subject's body. The instructionsmay be pre-recorded and determined by comparing the current positioningof the ultrasound device relative to one or more prior positions of theultrasound device which yielded the target ultrasound image.

B. Determining how to Guide an Operator of an Ultrasound Device toCapture a Medically Relevant Ultrasound Image

The disclosure provides techniques for guiding an operator of anultrasound device to capture a medically relevant ultrasound image of asubject. Teaching an individual how to perform a new task, such as howto use an ultrasound device, is a challenging endeavor. The individualmay become frustrated if they are provided instructions that are toocomplex or confusing. Accordingly, certain disclosed embodiments relateto new techniques for providing clear and concise instructions to guidethe operator of an ultrasound device to capture an ultrasound imagecontaining a target anatomical view. In some embodiments, the operatormay position the ultrasound device on the subject and a computing device(such as a mobile smartphone or a tablet) may generate a guidance planfor how to guide the operator to move the ultrasound device from aninitial position on the subject to a target position on the subject. Theguidance plan may comprise a series of simple instructions or steps(e.g., “MOVE UP,” “MOVE DOWN,” “MOVE LEFT,” or “MOVE RIGHT”) to guidethe operator from the initial position to the target position.

The guidance plan may optionally avoid using more complex instructionsthat may confuse the operator such as instructing the operator to movethe ultrasound device diagonally. Once the guidance plan has beengenerated, instructions from the guidance plan may be provided in aserial fashion to the operator to avoid overloading the operator withinformation. Thereby, the operator may easily follow the sequence ofsimple instructions to capture an ultrasound image containing the targetanatomical view.

In one embodiment, the guidance plan may be devised by comparing thecurrent ultrasound image with the target ultrasound image and bydetermining how the positioning of the ultrasound device with respect tothe subject should be changed to approach the target ultrasound image.

In some embodiments, techniques for determining how to guide an operatorof an ultrasound device to capture an ultrasound image containing atarget anatomical view may be embodied as a method that is performed by,for example, a computing device that is communicatively coupled to anultrasound device. The method may include obtaining an ultrasound imageof a subject captured using the ultrasound device. For example, thecomputing device may communicate with the ultrasound device to generateultrasound data and send the generated ultrasound data to the computingdevice. The computing device may, in turn, use the received ultrasounddata to generate the ultrasound image.

In some embodiments, the method may further include generating, usingthe ultrasound image, a guidance plan for how to guide the operator tocapture an ultrasound image of the subject containing the targetanatomical view when a determination is made that the ultrasound imagedoes not contain the target anatomical view. The guidance plan maycomprise, for example, a guide path along which an operator may beguided between an initial position of the ultrasound device on thesubject and a target position of the ultrasound device on the subjectwhere an ultrasound image that contains the target anatomical view maybe captured. For example, the initial position of the ultrasound devicemay be identified using the ultrasound image and the target position ofthe ultrasound device may be identified using the target anatomicalview. Once the initial and target positions of the ultrasound devicehave been identified, a guide path may be determined between the twopositions.

In some embodiments, the guide path between the initial position of theultrasound device and the target position of the ultrasound device maynot be the most direct path between the two positions. For example, alonger guide path may be selected that forms an “L” over a directdiagonal line between the two points because the “L” shaped guide pathmay be easier to communicate to an operator. In some embodiments, theguide path may advantageously minimize travel of the ultrasound deviceover areas of the subject that contain hard tissue (e.g., bone).Capturing an ultrasound image of bone may yield a blank (or nearlyblank) ultrasound image because the acoustic waves emitted by anultrasound device typically do not penetrate hard tissues. As a result,there may be little or no information contained in the ultrasound imagethat may be used by the computing device to determine a position of theultrasound device on the subject. Minimizing travel over these hardtissues may advantageously allow the computing device to more easilytrack the progress of the ultrasound device as the operator moves theultrasound device along the guide path by analyzing the capturedultrasound images.

The method may further include providing at least one instruction to theoperator based on the determined guidance plan. For example,instructions may be generated that instruct the operator to move theultrasound device along a determined guide path in the guidance plan.Alternatively (or additionally), the guidance plan may include asequence of instructions to guide the operator of the ultrasound deviceto move the device and the instructions may be provided directly fromthe guidance plan. The instructions may be provided from the guidanceplan in a serial fashion (e.g., one at a time).

It should be appreciated that the guidance plan may be updated (e.g.,continuously updated) based on the actions actually taken by anoperator. In some embodiments, the guidance plan may be updated when theaction taken by an operator does not match the instruction provided tothe operator. For example, the computing device may issue an instructionfor the operator to move the ultrasound device left and the operator mayhave inadvertently moved the ultrasound device up. In this example, thecomputing device may generate a new guidance plan between the currentposition of the ultrasound device and the target position of theultrasound device.

C. Creating an Augmented Reality Interface to Guide an Operator of anUltrasound Device

The disclosure provides techniques for creating an augmented realityinterface that guides an operator of an ultrasound device. Providingwritten and/or spoken instructions may be challenging for an operator tounderstand. For example, conveying an instruction to move an ultrasounddevice in a particular direction (e.g., “MOVE LEFT”) may be ambiguousbecause the point of reference used by the operator may be different.Thereby, the operator may move the ultrasound device in an incorrectdirection while believing that they are properly following theinstructions. Accordingly, certain disclosed embodiments relate to newtechniques for providing instructions to an operator of an ultrasounddevice through an augmented reality interface. In the augmented realityinterface, the instructions may be overlaid onto a view of theoperator's real-world environment. For example, the augmented realityinterface may include a view of the ultrasound device positioned on thesubject and an arrow indicative of the particular direction that theultrasound device should be moved. Thereby, an operator may easilyre-position the ultrasound device by moving the ultrasound device on thesubject in a direction consistent with the arrow in the augmentedinterface.

In some embodiments, techniques for providing an augmented realityinterface to guide an operator to capture an ultrasound image containinga target anatomical view may be embodied as a method that is performedby, for example, a computing device having (or being in communicationwith) a non-acoustic imaging device such as an imaging device configuredto detect light. The method may include capturing, using a non-acousticimaging device, an image of the ultrasound device. For example, an imagemay be captured of the ultrasound device positioned on a subject.

In some embodiments, the method may further include generating acomposite image at least in part by overlaying, onto the image of theultrasound device, at least one instruction indicating how the operatoris to reposition the ultrasound device. For example, a pose (e.g.,position and/or orientation) of the ultrasound device in the capturedimage may be identified using an automated image processing technique(e.g., a deep learning technique) and the information regarding the poseof the ultrasound device may be used to overlay an instruction onto atleast part of the ultrasound device in the captured image. Exampleinstructions that may be overlaid onto the image of the ultrasounddevice include symbols (such as arrows) indicating a direction in whichthe operator is to move the device.

It should be appreciated that additional elements may be overlaid ontothe image of the ultrasound device using the identified pose of theultrasound device. For example, the ultrasound image captured using theultrasound device may be overlaid onto the image of the ultrasounddevice in such a fashion to make the ultrasound image appear as thoughit is extending outward from the ultrasound device into the subject.Thereby, the operator may gain a better appreciation for the particularregion of the subject that is being imaged given the current position ofthe ultrasound device on the subject.

In some embodiments, the method may further include presenting thecomposite image to the operator. For example, the computing device mayinclude an integrated display and the composite image may be displayedto the operator using the display.

D. Tracking a Location of an Ultrasound Device Using a Marker on theUltrasound Device

The disclosure provides techniques for tracking a location of anultrasound device using a marker disposed on the ultrasound device. Asdiscussed above, providing instructions to an operator of an ultrasounddevice through an augmented reality interface may make the instructionsclearer and easier to understand. The augmented reality interface mayinclude a captured image of a real-world environment (e.g., captured bya camera on a mobile smartphone) and one or more instructions overlaidonto the captured image regarding how to move the ultrasound device.Such augmented reality interfaces may be even more intuitive when theinstructions are positioned relative to real-world objects in a capturedimage. For example, an arrow that instructs the operator to move theultrasound device left may be clearer to the operator when the arrow ispositioned proximate the ultrasound device in the captured image.Accordingly, aspects of the technology described herein relate to newtechniques for tracking an ultrasound device in a captured image suchthat instructions may be properly positioned in the augmented realityinterface. The problem of identifying the location of the ultrasounddevice in a captured image may be eased by placing a distinct marker onthe ultrasound device that is visible in the captured image. The markermay have, for example, a distinctive pattern, color, and/or image thatmay be readily identified using automated image processing techniques(such as deep learning techniques). Thereby, the position of theultrasound device in the captured image may be identified by locatingthe marker in the captured image. Once the position of the ultrasounddevice in the captured image has been identified, an instruction may beoverlaid onto the captured image at a position proximate the ultrasounddevice to form a more intuitive augmented reality interface.

In some embodiments, techniques for tracking a location of an ultrasounddevice in a captured image using a marker disposed on the ultrasounddevice may be embodied as a method that is performed by, for example, acomputing device that is communicatively coupled to an ultrasounddevice. The location of the ultrasound device in a captured image may betracked to, for example, properly position an instruction over thecaptured image so as to form an augmented reality interface. Forexample, the instruction may be positioned proximate the ultrasounddevice in the captured image. In some embodiments, these techniques maybe embodied as a method that is performed by, for example, a computingdevice having (or in communication with) a non-acoustic imaging devicesuch as an imaging device configured to detect light. The non-acousticimaging device may be employed to capture an image of a marker on anultrasound device. The marker may be constructed to have a distinctivepattern, color, and/or image that may be recognized. The marker may beimplemented in any of a variety of ways. For example, the marker may be:a monochrome marker, a holographic marker, and/or a dispersive marker.Monochrome markers may comprise a monochrome pattern such as ArUcomarkers. Holographic markers may comprise a hologram that presentsdifferent images depending upon the particular angle from which thehologram is viewed. Dispersive markers may comprise a dispersive elementthat presents different colors depending upon the particular angle fromwhich the dispersive element is viewed.

In some embodiments, the method may further include automaticallyidentifying a pose of the ultrasound device at least in part byanalyzing at least one characteristic of the marker in the capturedimage. For example, a location of the marker in the image may beidentified to determine a position of the ultrasound device in theimage. Additionally (or alternatively), one or more properties of themarker may be analyzed to determine an orientation of the ultrasounddevice in the image. For example, the marker may be a dispersive markerand the color of the marker may be analyzed to determine an orientationof the ultrasound device. In another example, the marker may be aholographic marker and the particular image presented by the marker maybe analyzed to determine an orientation of the ultrasound device.

In some embodiments, the method may further include providing aninstruction to an operator of the ultrasound device using the identifiedpose of the ultrasound device. For example, the instruction may comprisea symbol (e.g., an arrow) overlaid onto the captured image that ispresented to the operator. In this example, the identified pose of theultrasound device in the image may be employed to accurately positionthe symbol over at least part of the ultrasound device in the capturedimage.

E. Automatically Interpreting Captured Ultrasound Images

The disclosure provides techniques for automatically interpretingcaptured ultrasound images to identify medical parameters of a subject.Novice operators of an ultrasound device may not be able to interpretcaptured ultrasound images to glean medically relevant information aboutthe subject. For example, a novice operator may not know how tocalculate medical parameters of the subject from a captured ultrasoundimage (such as an ejection fraction of a heart of the subject).Accordingly, certain disclosed embodiments relate to new techniques forautomatically analyzing a captured ultrasound image to identify suchmedical parameters of the subject. In some embodiments, the medicalparameters may be identified using state of the art image processingtechnology such as deep learning. For example, deep learning techniquesmay be employed to identify the presence of particular organs (such as aheart or a lung) in the ultrasound image. Once the organs in ultrasoundimage have been identified, the characteristics of the organs (e.g.,shape and/or size) may be analyzed to determine a medical parameter ofthe subject (such as an ejection fraction of a heart of the subject).

In some embodiments, techniques for identifying a medical parameter of asubject using a captured ultrasound image may be embodied as a methodthat is performed by, for example, a computing device that iscommunicatively coupled to an ultrasound device. The method may includeobtaining an ultrasound image of a subject captured using an ultrasounddevice. For example, the computing device may communicate with theultrasound device to generate ultrasound data and send the generatedultrasound data to the computing device. The computing device may, inturn, use the received ultrasound data to generate the ultrasound image.

In some embodiments, the method may further include identifying ananatomical feature of the subject in the ultrasound image using anautomated image processing technique. Example anatomical features of thesubject that may be identified include: a heart ventricle, a heartvalve, a heart septum, a heart papillary muscle, a heart atrium, anaorta, and a lung. These anatomical features may be identified using anyof a variety of automated image processing techniques such as deeplearning techniques.

In some embodiments, the method may further include identifying amedical parameter of the subject using the identified anatomical featurein the ultrasound image. For example, an ultrasound image of a heart maybe captured and the ventricle in the ultrasound image may be identifiedas an anatomical feature. In this example, one or more dimensions of theheart ventricle may be calculated using the portion of the ultrasoundimage identified as being a heart ventricle to identify medicalparameters associated with the heart. Example medical parametersassociated with the heart include: an ejection fraction, a fractionalshortening, a ventricle diameter, a ventricle volume, an end-diastolicvolume, an end-systolic volume, a cardiac output, a stroke volume, anintraventricular septum thickness, a ventricle wall thickness, and apulse rate.

F. Automatically Generating a Diagnosis of a Medical Condition

The disclosure provides techniques for generating a diagnosis of amedical condition of a subject using a captured ultrasound image. Noviceoperators of an ultrasound device may be unaware of how to use anultrasound device to diagnose a medical condition of the subject. Forexample, the operator may be unsure of which anatomical view of asubject to image to diagnose the medical condition. Further, theoperator may be unsure of how to interpret a captured ultrasound imageto diagnose the medical condition. Accordingly, certain disclosedembodiments relate to new techniques for assisting an operator of anultrasound device to diagnose a medical condition of a subject. In someembodiments, these techniques may be employed in a diagnostic App thatmay be installed on a computing device (e.g., a smartphone) of a healthcare professional. The diagnostic App may walk the health careprofessional through the entire process of diagnosing a medicalcondition of the subject. For example, the diagnostic App may prompt thehealth care professional for medical information about the subject(e.g., age, weight, height, resting heart rate, blood pressure, bodysurface area, etc.) that may be employed to determine a particularanatomical view of the subject to image with an ultrasound device. Then,the diagnostic App may guide the health care professional to capture anultrasound image of the anatomical view. The diagnostic App may employthe captured ultrasound image (or sequence of ultrasound images) and/orraw ultrasound data from the ultrasound device. It should be appreciatedthat other information (such as the medical information about thesubject) may be employed in combination with the ultrasound image(s)and/or raw ultrasound data to diagnose the medical condition of thesubject.

In some embodiments, techniques for diagnosing a medical condition of asubject using an ultrasound device may be embodied as a method that isperformed by, for example, a computing device that is communicativelycoupled to an ultrasound device. The method may include receivingmedical information about a subject. Example medical information about asubject includes: heart rate, blood pressure, body surface area, age,weight, height, and medication being taken by the subject. The medicalinformation may be received from an operator by, for example, posing oneor more survey questions to the operator. Alternatively (oradditionally), the medical information may be obtained from an externaldevice such as an external server.

In some embodiments, the method may further include identifying a targetanatomical view of the subject to be captured using an ultrasound devicebased on the received medical information. Example anatomical views thatmay be identified include: a parasternal long axis (PLAX) anatomicalview, a parasternal short-axis (PSAX) anatomical view, an apicalfour-chamber (A4C) anatomical view, and an apical long axis (ALAX)anatomical view. In some embodiments, the medical information may beanalyzed to determine whether the subject has any health problemsassociated with a particular organ that may be imaged, such as a heartor a lung. If the medical information indicated that the subject hassuch health problems, an anatomical view associated with the organ maybe identified. For example, the medical information may include anindication that the subject has symptoms of congestive heart failure(such as recently experiencing paroxysmal nocturnal dyspnea). In thisexample, an anatomical view associated with the heart (such as the PLAXanatomical view) may be identified as the appropriate view to becaptured.

In some embodiments, the method may further include obtaining anultrasound image containing the target anatomical view of the subject.For example, the ultrasound image may be obtained from an electronichealth record of the subject. Additionally (or alternatively), theoperator may be guided to obtain the ultrasound image containing thetarget anatomical view. For example, the operator may be provided one ormore instructions (e.g., a sequence of instruction) to reposition theultrasound device on the subject such that the ultrasound device isproperly positioned on the subject to capture the target anatomicalview.

In some embodiments, the method may further include generating adiagnosis of a medical condition of the subject using the ultrasoundimage containing the target anatomical view. For example, one or moremedical parameters (e.g., an ejection fraction) may be extracted fromthe ultrasound image (or sequence of ultrasound images) and employed togenerate a diagnosis. It should be appreciated that additionalinformation separate from the ultrasound image containing the targetanatomical view may be employed to identify a diagnosis of a medicalcondition of the subject. For example, the medical information regardingthe subject may be employed in combination with one or more medicalparameters extracted from the ultrasound device to generate thediagnosis.

In some embodiments, the method may further include generating one ormore recommended treatments for the subject. The recommended treatmentsmay be generated based on diagnosed medical condition of the subject.For example, the subject may be diagnosed with a heart condition (e.g.,congestive heart failure) and the recommended treatment may comprise apharmaceutical drug employed to treat the heart condition (e.g., a betablocker drug).

G. Further Description

FIG. 1 shows an example ultrasound system 100 that is configured toguide an operator of an ultrasound device 102 to obtain an ultrasoundimage of a target anatomical view of a subject 101. As shown, theultrasound system 100 comprises an ultrasound device 102 that iscommunicatively coupled to the computing device 104 by a communicationlink 112. The computing device 104 may be configured to receiveultrasound data from the ultrasound device 102 and use the receivedultrasound data to generate an ultrasound image 110. The computingdevice 104 may analyze the ultrasound image 110 to provide guidance toan operator of the ultrasound device 102 regarding how to reposition theultrasound device 102 to capture an ultrasound image containing a targetanatomical view. For example, the computing device 104 may analyze theultrasound image 110 to determine whether the ultrasound image 110contains a target anatomical view, such as a PLAX anatomical view. Ifthe computing device 104 determines that the ultrasound image 110contains the target anatomical view, the computing device 104 mayprovide an indication to the operator using a display 106 that theultrasound device 102 is properly positioned. Otherwise, the computingdevice 104 may provide an instruction 108 using the display 106 to theoperator regarding how to reposition the ultrasound device 102.

The ultrasound device 102 may be configured to generate ultrasound data.The ultrasound device 102 may be configured to generate ultrasound databy, for example, emitting acoustic waves into the subject 101 anddetecting the reflected acoustic waves. The detected reflected acousticwave may be analyzed to identify various properties of the tissuesthrough which the acoustic wave traveled, such as a density of thetissue. The ultrasound device 102 may be implemented in any of varietyof ways. For example, the ultrasound device 102 may be implemented as ahandheld device (as shown in FIG. 1) or as a patch that is coupled topatient using, for example, an adhesive. Example ultrasound devices aredescribed in detail below in the Example Ultrasound Devices section.

The ultrasound device 102 may transmit ultrasound data to the computingdevice 104 using the communication link 112. The communication link 112may be a wired (or wireless) communication link. In some embodiments,the communication link 112 may be implemented as a cable such as aUniversal Serial Bus (USB) cable or a Lightning cable. In theseembodiments, the cable may also be used to transfer power from thecomputing device 104 to the ultrasound device 102. In other embodiments,the communication link 112 may be a wireless communication link such asa BLUETOOTH, WiFi, or ZIGBEE wireless communication link.

The computing device 104 may comprise one or more processing elements(such as a processor) to, for example, process ultrasound data receivedfrom the ultrasound device 102. Additionally, the computing device 104may comprise one or more storage elements (such as a non-transitorycomputer readable medium) to, for example, store instructions that maybe executed by the processing element(s) and/or store all or any portionof the ultrasound data received from the ultrasound device 102. Itshould be appreciated that the computing device 104 may be implementedin any of a variety of ways. For example, the computing device 104 maybe implemented as a mobile device (e.g., a mobile smartphone, a tablet,or a laptop) with an integrated display 106 as shown in FIG. 1. In otherexamples, the computing device 104 may be implemented as a stationarydevice such as a desktop computer. Additional example implementations ofthe computing device are described below in the Example UltrasoundSystems section.

The computing device 104 may be configured to provide guidance to anoperator of the ultrasound device 102 using the ultrasound data receivedfrom the ultrasound device 102. In some embodiments, the computingdevice 104 may generate the ultrasound image 110 using the receivedultrasound data and analyze the ultrasound image 110 using an automatedimage processing technique to generate the instruction 108 regarding howthe operator should re-position the ultrasound device 102 to capture anultrasound image containing the target anatomical view. For example, thecomputing device 104 may identify the anatomical view contained in theultrasound image 110 using a machine learning technique (such as a deeplearning technique) and determine whether the anatomical view containedin the ultrasound image 110 matches the target anatomical view. If theidentified anatomical view matches the target anatomical view, thecomputing device 104 may provide an indication that the ultrasound isproperly positioned via the display 106. Otherwise, the computing device104 may identify an instruction to provide the operator to repositionthe ultrasound device 102 and provide the instruction via the display106. In another example, the computing device 104 may generate theinstruction 108 without performing the intermediate step of determiningwhether the ultrasound image 110 contains the target anatomical view.For example, the computing device 104 may use a machine learningtechnique (such as a deep learning technique) to directly map theultrasound image 110 to an output to provide to the user such as anindication of proper positioning or an instruction to reposition theultrasound device 102 (e.g., instruction 108).

In some embodiments, the computing device 104 may be configured togenerate the instruction 108 for the operator regarding how to positionthe ultrasound device 102 on the subject 101 using a guidance plan. Theguidance plan may comprise a guide path indicative of how the operatorshould be guided to move the ultrasound device 102 from an initialposition on the subject 101 to a target position on the subject 101where an ultrasound image containing the target anatomical view may becaptured. An example of such a guide path on a subject is shown in FIG.2. As shown, the ultrasound device may be initially positioned on asubject 201 at an initial position 202 (on a lower torso of the subject201) and the computing device may generate a guide path 208 between theinitial position 202 and a target position 204. The guide path 208 maybe employed by the computing device to generate a sequence ofinstructions to provide the operator. For example, the computing devicemay generate a first instruction to “MOVE RIGHT” and a secondinstruction to “MOVE UP” for the guide path 208. The generatedinstructions may also include an indication of the magnitude of themovement, such as “MOVE RIGHT 5 CENTIMETERS.” The computing device mayprovide these instructions serially (e.g., one at a time) to avoidoverloading the operator with information.

The computing device may identify the initial position 202 by analyzingthe ultrasound data received from the ultrasound device using anautomated image processing technique (e.g., a deep learning technique).For example, the computing device may provide an ultrasound image(generated using the ultrasound data) as an input to a neural networkthat is configured (e.g., trained) to provide as an output an indicationof the anatomical view contained in the ultrasound image. Then, thecomputing device may map the identified anatomical view to a position onthe subject 201. The mappings between anatomical views and positions onthe subject 201 may be, for example, stored locally on the computingdevice.

The computing device may identify the target position 204 based on thetarget anatomical view. For example, the computing device may map thetarget anatomical view to a position on the subject 201. The mappingsbetween target anatomical views and positions on the subject 201 may be,for example, stored locally on the computing device.

Once the initial position 202 and the target position 204 have beenidentified, the computing device may identify the guide path 208 that anoperator should follow to move the ultrasound device from the initialposition 202 to the target position 204. The computing device maygenerate the guide path 208 by, for example, identifying a shortest pathbetween the initial position 202 and the target position 204 (e.g., adiagonal path). Alternatively, the computing device may generate theguide path 208 by identifying a shortest path between the initialposition 202 and the target position 204 that satisfies one or moreconstraints. The one or more constraints may be selected to, forexample, ease communication of instructions to the operator to move theultrasound device along the guide path 208. For example, movement inparticular directions (such as diagonal directions) may be morechallenging to accurately communicate to an operator. Thereby, thecomputing device may identify a shortest path that omits diagonalmovements as the guide path as shown by the “L” shaped guide path 208 inFIG. 2. Additionally (or alternatively), the guide path 208 may beselected to minimize traversal over hard tissue (e.g., bone) in thesubject. Minimizing the travel over such hard tissues may advantageouslyallow the computing device to more readily track the movement of theultrasound device along the guide path 208. For example, ultrasoundimages of bone may be blank (or nearly blank) because the acoustic wavesemitted by an ultrasound device typically do not penetrate hard tissues.The computing device may be unable to analyze such ultrasound images todetermine which anatomical view they belong to and, thereby, lose trackof the position of the ultrasound device on the subject 201. Minimizingtravel over these hard tissues may advantageously allow the computingdevice to more easily track the progress of the ultrasound device as theoperator moves the ultrasound device along the guide path 208 byanalyzing the captured ultrasound images.

The computing device may store the generated guide path 208 locally anduse the guide path to generate a sequence of instructions to provide tothe operator. For example, the computing device may use the guide path208 to generate the sequence of instructions: (1) “MOVE LATERAL,” (2)“MOVE UP,” and (3) “TWIST CLOCKWISE.” These instructions may be, inturn, provided to the operator in a serial fashion to guide the operatorto move the ultrasound device from the initial position 202 to thetarget position 204.

As discussed above, novice operators of an ultrasound device may havelittle or no knowledge of human physiology. Thereby, the initialposition 202 may be far away from the target position 204. For example,an operator may initially place the ultrasound device on a leg of thesubject 201 when the target position 204 is on an upper torso of thesubject 201. Providing a sequence of individual instructions to move theultrasound device from the distant initial position 202 to the targetposition 204 may be a time-consuming process. Accordingly, the computingdevice may initially provide the operator a coarse instruction to movethe ultrasound device to a general area of the subject 201 (such as anupper torso of the subject 201) and subsequently provide one or morefine instructions to move the ultrasound device in particular directions(such as “MOVE UP”).

In some embodiments, the computing device may make the determination asto whether to issue a coarse instruction or a fine instruction based ona determination as to whether the ultrasound device is positioned on thesubject within a predetermined area 206 on the subject 201. Thepredetermined area 206 may be an area on the subject 201 that includesthe target position 204 and is easy for the operator to identify. Forexample, the target position 204 may be over a heart of the subject 201and the predetermined area 206 may comprise an upper torso of thesubject. The computing device may provide a fine instruction responsiveto the position of the ultrasound device being within the predeterminedarea 206 and provide a coarse instruction responsive to the ultrasounddevice being outside of the predetermined area 206. For example, anoperator may initially position the ultrasound device on a leg of thesubject 201 and the computing device may provide a coarse instructionthat instructs the operator to move the ultrasound device to an uppertorso (e.g., the predetermined area 206) of the subject 201. Once theoperator has positioned the ultrasound device on the upper torso of thesubject 201 (and thereby within the predetermined area 206), thecomputing device may provide a fine instruction including an indicationof a particular direction to move the ultrasound device towards thetarget position 204.

Providing coarse instructions may advantageously expedite the process ofguiding the operator of the ultrasound device. For example, an operatormay be unfamiliar with human physiology and initially place theultrasound device on a leg of the subject 201 while the operator isattempting to capture an ultrasound image containing an anatomical viewof a heart of the subject 201. In this example, the operator may beprovided a coarse instruction including an indication of where to placethe ultrasound device (e.g., on an upper torso of the subject) insteadof providing a set of instructions for the operator to move theultrasound device: (1) from the thigh to the lower torso and (2) fromthe lower torso to the upper torso.

FIG. 3A shows an example coarse instruction 302 that may be provided toan operator via a display 306 on a computing device 304. The coarseinstruction 302 may be provided when the ultrasound device is positionedoutside of a predetermined area on the subject. As shown, the coarseinstruction 302 includes an indication of where the operator shouldposition the ultrasound device on the subject to be within thepredetermined area. In particular, the coarse instruction 302 comprisesa symbol 308 (e.g., a star) showing where the predetermined region islocated on a graphical image of the subject 301. The coarse instruction302 also includes a message 310 with an arrow pointing to the symbol 308instructing the operator to “POSITION ULTRASOUND DEVICE HERE” tocommunicate to the operator that the ultrasound device should be placedwhere the symbol 308 is located on the graphical image of the subject301.

FIG. 3B shows an example fine instruction 312 that may be provided to anoperator via the display 306 on the computing device 304. The fineinstruction 312 may be provided when the ultrasound device is positionedwithin the predetermined area on the subject. As shown, the fineinstruction 312 includes a symbol 314 indicating which direction theoperator should move the ultrasound device. The symbol 314 may beanimated in some implementations. For example, the symbol 314 (e.g., anarrow and/or model of the ultrasound device) may move in a direction inwhich the ultrasound device is to be moved. The fine instruction 312 mayalso comprise a message 316 that compliments the symbol 314 such as themessage “TURN CLOCKWISE.” The symbol 314 and/or the message 316 may beoverlaid onto a background image 311. The background image 311 may be,for example, an ultrasound image generated using ultrasound datareceived from the ultrasound device.

FIG. 3C shows an example confirmation 318 that may be provided to anoperator via the display 306 on the computing device 304. Theconfirmation 318 may be provided when the ultrasound device is properlypositioned on the subject to capture an ultrasound image containing thetarget anatomical view. As shown, the confirmation 318 includes a symbol320 (e.g., a checkmark) indicating that the ultrasound device isproperly positioned. The confirmation 318 may also comprise a message322 that compliments the symbol 320 such as the message “HOLD.” Thesymbol 320 and/or the message 322 may be overlaid onto the backgroundimage 311. The background image 311 may be, for example, an ultrasoundimage generated using ultrasound data received from the ultrasounddevice.

Once the operator has successfully captured an ultrasound image thatcontains the target anatomical view, the computing device may beconfigured to analyze the captured ultrasound image. For example, thecomputing device may analyze the captured ultrasound image using anautomated image processing technique to identify a medical parameter ofthe subject. Example medical parameters of the subject that may beobtained from the ultrasound image include: an ejection fraction, afractional shortening, a ventricle diameter, a ventricle volume, anend-diastolic volume, an end-systolic volume, a cardiac output, a strokevolume, an intraventricular septum thickness, a ventricle wallthickness, and a pulse rate. The computing device may identify thesemedical parameters by, for example, identifying an anatomical feature inthe ultrasound image (such as a heart ventricle, a heart valve, a heartseptum, a heart papillary muscle, a heart atrium, an aorta, and a lung)and analyzing the identified anatomical feature. The computing devicemay identify the anatomical feature using an automated imagingprocessing technique (such as a deep learning technique). For example,the computing device may provide the captured ultrasound image to aneural network that is configured (e.g., trained) to provide as anoutput an indication of which pixels in the ultrasound image areassociated with a particular anatomical feature. It should beappreciated that this neural network may be separate and distinct fromany neural networks employed to guide the operator.

The generated medical parameters may be overlaid onto the capturedultrasound image as shown in FIG. 4. As shown, a computing device 404may display (via an integrated display 406) an ultrasound image 408 anda set of medical parameters 410 overlaid onto the ultrasound image 408.The ultrasound image 408 may contain a PLAX view of a subject thatincludes a view of a heart of the subject. In the ultrasound image 408,the computing device may identify the left ventricle as an anatomicalfeature 402 and analyze the characteristics of the left ventricle (suchas the left ventricle diameter shown as anatomical featurecharacteristic 404) to identify the medical parameters 410. The medicalparameters 410 shown in FIG. 4 comprise: a left ventricle diameter (LVD)of 38.3 millimeters (mm), a left ventricle end-systolic diameter (LVESD)of 38.2 mm, a left ventricle end-diastolic diameter (LVEDD) of 49.5 mm,a fractional shortening (FS) of 23%, an ejection fraction (EF) of 45%.

It should be appreciated that the computing device may identify themedical parameters 410 using more than a single ultrasound imagecontaining the target anatomical view. In some embodiments, a sequenceof ultrasound images of the heart may be captured that span at least onecomplete heartbeat to generate the medical parameters. For example, theultrasound images may be analyzed to determine which ultrasound imagewas captured at the end of the contraction of a heart ventricle(referred to as the end-systolic image) and which ultrasound image wascaptured just before the start of the contraction of a heart ventricle(referred to as the end-diastolic image). The end-systolic image may beidentified by, for example, identifying the ultrasound image in thesequence that has a smallest ventricle volume (or diameter). Similarly,the end-diastolic image may be identified by, for example, identifyingthe ultrasound image in the sequence that has the largest ventriclevolume (or diameter). The end-systolic image may be analyzed todetermine one or more medical parameters that are measured at the end ofthe heart contraction such as an end-systolic diameter (ESD) and/or anend-systolic volume (ESV). Similarly, the end-diastolic image may beanalyzed to determine one or more medical parameters that are measuredjust before the start of a heart contraction such as an end-diastolicdiameter (EDD) and/or an end-diastolic volume (EDV). Some medicalparameters may require analysis of both the end-systolic image and theend-diastolic image. For example, the identification of the EF mayrequire (1) an EDV identified using the end-diastolic image and (2) anESV identified using the end-systolic image as shown in Equation (1)below:

$\begin{matrix}{{EF} = {\frac{{EDV} - {ESV}}{EDV}*100}} & (1)\end{matrix}$Similarly, the identification of the FS may require (1) an EDDidentified using the end-diastolic image and (2) an ESD identified usingthe end-systolic image as shown in Equation (2) below:

$\begin{matrix}{{FS} = {\frac{{EDD} - {ESD}}{EDD}*100}} & (2)\end{matrix}$

In some embodiments, the computing device may change a color of themedical parameters 410 shown in the display 406 based on the value ofthe medical parameters. For example, the medical parameters 410 may bedisplayed in a first color (e.g., green) to indicate that the values arewithin a normal range, a second color (e.g., orange) to indicate thatthe values are in a borderline abnormal range, and a third color (e.g.,red) to indicate that the values are in an abnormal range.

Example Augmented Reality Interfaces

The inventors have recognized that providing instructions to an operatorthrough an augmented reality interface may advantageously make theinstructions easier to understand for the operator. FIG. 5A shows anexample ultrasound system that is configured to provide the operator anaugmented reality interface. As shown, the ultrasound system comprisesan ultrasound device 502 communicatively coupled to a computing device504 via a communication link 512. The ultrasound device 502,communication link 512, and/or computing device 504 may be similar to(or the same as) the ultrasound device 102, the communication link 112,and/or the computing device 104, respectively, described above withreference to FIG. 1. The ultrasound system further comprises a marker510 disposed of the ultrasound device 502. The marker advantageouslyallow the computing device 504 to more easily track the location of theultrasound device in non-acoustic images captured by an imaging device506 (e.g., integrated into the computing device 504). The computingdevice 504 may use the tracked location of the ultrasound device in thenon-acoustic images to overlay one or more elements (e.g., instructions)onto the non-acoustic images to form an augmented reality interface.Such an augmented reality interface may be displayed via a display 508(e.g., integrated into the computing device 502 and disposed on anopposite side relative to the imaging device 506).

It should be appreciated that the computing device 504 does not need tobe implemented as a handheld device. In some embodiments, the computingdevice 504 may be implemented as a wearable device with a mechanism todisplay instructions to an operator. For example, the computing device504 may be implemented as a wearable headset and/or a pair of smartglasses (e.g., GOOGLE GLASS, APPLE AR glasses, and MICROSOFT HOLOLENS).

FIG. 5B shows another view of the ultrasound system from the perspectiveof an operator. As shown, the display 508 in the computing device 504displays an augmented reality interface comprising a non-acoustic image512 of the ultrasound device 502 being used on the subject 501 (e.g.,captured by the imaging device 506) and one or more elements overlaidonto the image 512. For example, an instruction 516 indicative of adirection for the operator to move the ultrasound device 502, a symbolindicating a location of the target anatomical plane, and/or anultrasound image 514 captured by the ultrasound device 502 may beoverlaid onto the image 512. These elements may be implemented, forexample, as: opaque elements (so as to obscure the portion of the image512 under the element), transparent elements (so as to not obscure theportion of the image 512 under the element), pseudo colorized elements,and/or cut-away elements.

In some embodiments, the instruction 516 may be overlaid onto the image512 such that at least a portion of the instruction 516 is overlaid ontothe ultrasound device 502 in the image 512. The computing device 504may, for example, use the marker 510 to identify a pose (e.g., aposition and/or orientation) of the ultrasound device 502 in the image512 and position the instruction 516 in the augmented reality interfaceusing the identified pose. The marker 510 may be constructed to have oneor more distinctive characteristics that may easily be recognized in theimage 512. Example markers include: monochrome markers, holographicmarkers, and dispersive markers. Monochrome markers may comprise amonochrome pattern such as ArUco markers. Holographic markers maycomprise a hologram that presents different images depending on theparticular angle from which the hologram is viewed. Dispersive markersmay comprise a dispersive element that presents different colorsdepending on the particular angle from which the dispersive element isviewed. The computing device 504 may identify the pose of the ultrasounddevice 502 in any of a variety of ways. In some embodiments, thecomputing device may identify a position of the ultrasound device 502 inthe image 512 by identifying a location of the marker 510. The locationof the marker 510 may be identified by searching for one or moredistinct characteristics of the marker 510 in the image 512.Additionally (or alternatively), the computing device may identify anorientation of the ultrasound device 502 in the image 512 by analyzingone or more characteristics of the marker 512. For example, the marker510 may be a dispersive marker and the computing device may identify anorientation of the ultrasound device 502 in the image 512 by identifyinga color of the marker 510 in the image 512. In another example, themarker 510 may be a holographic marker and the computing device mayidentify an orientation of the ultrasound device 502 in the image 512 byidentifying an image presented by the marker 510 in the image 512. Inyet another example, the marker 510 may be a patterned monochrome markerand the computing device may identify an orientation of the ultrasounddevice 502 in the image 512 by identifying an orientation of the patternon the marker 510 in the image 512.

It should be appreciated that the pose of the ultrasound device 502 maybe identified without the marker 510. For example, the ultrasound device502 may have distinctive characteristics (e.g., shape and/or color) thatmay be readily identifiable in the image 512. Thereby, the computingdevice 504 may identify the pose of the ultrasound device 502 in theimage 510 by analyzing one or more characteristics of the ultrasounddevice 502 in the image 510.

In some embodiments, the identified pose of the ultrasound device 502 inthe image 512 may be employed to overlay other elements onto the image512 separate from the instruction 516. For example, the identified poseof the ultrasound device 502 may be employed to overlay the ultrasoundimage 514 over the image 512 such that ultrasound image 514 appears tobe extending out of the ultrasound device 502 into the subject 501. Sucha configuration may advantageously provide an indication to the operatorof the particular portion of the subject that is being imaged by theultrasound device 502. An example of such an augmented reality interfaceis shown in FIG. 6 being displayed on a display 606 of a computingdevice 604. The augmented reality interface overlays the ultrasoundimage 610 and an ultrasound device symbol 608 onto an image of anultrasound device 602 being used to image the subject 601 (e.g.,captured from a front-facing camera in the handheld device computingdevice 604). As shown, the ultrasound image 610 is overlaid onto theportion of the subject 601 that is being imaged by the ultrasound device602. In particular, the ultrasound image 610 has been positioned andoriented so as to be extending from the ultrasound device 602 into thesubject 601. This position and orientation of the ultrasound image 610may indicate to the operator the particular portion of the subject 601that is being imaged. For example, the ultrasound device 602 may bepositioned on an upper torso of the subject 601 and the ultrasound image610 may extend from an end of the ultrasound device 602 in contact withthe subject 601 into the upper torso of the subject 601. Thereby, theoperator may be informed that the captured image is that of a 2Dcross-section of body tissue in the upper torso of subject 601.

It should be appreciated that additional (or fewer) elements may beoverlaid onto the image of the ultrasound device 602 being used on thesubject 601 in FIG. 6. For example, the ultrasound device symbol 608overlaid onto the ultrasound device 602 may be omitted. Additionally (oralternatively), the user interface may overlay instructions (e.g.,augmented reality arrows) onto the image of the ultrasound device 602 onthe subject 601 to provide guidance to the operator.

Example Diagnostic Applications

The inventors have recognized that ultrasound imaging techniques may beadvantageously combined with diagnostics and treatment recommendationsto provide an ecosystem of intelligent and affordable products andservices that democratize access to medical imaging and accelerateimaging into routine clinical practice and/or patient monitoring. Thismay provide an advance in conventional clinical decision support (CDS)applications by empowering healthcare professionals and/or patients tomake diagnostic and treatment decisions at an earlier state of disease,as well as to assist novice imaging users (e.g., consumers) to detectvarious conditions earlier and monitor patient response to therapy.

The technology improvements described herein may enable, among othercapabilities, focused diagnosis, early detection and treatment ofconditions by an ultrasound system. The ultrasound system may comprisean ultrasound device that is configured to capture ultrasound images ofthe subject and a computing device in communication with the ultrasounddevice. The computing device may execute a diagnostic application thatis configured to perform, for example, one or more of the followingfunctions: (1) acquire medical information regarding the subject, (2)identify an anatomical view of the subject to image with the ultrasounddevice based on the acquired medical information regarding the subject,(3) guide the operator to capture ultrasound image(s) that contain theidentified anatomical view, (4) provide a diagnosis (or pre-diagnosis)of a medical condition of the subject based on the captured ultrasoundimages, and (5) provide one or more recommended treatments based on thediagnosis.

FIGS. 7A-7H show an example user interface for a diagnostic applicationthat is configured to assist an operator determine whether a subject isexperiencing heart failure. The diagnostic application may be designedto be used by, for example, a health care professional such as a doctor,a nurse, or a physician assistant. The diagnostic application may beexecuted by, for example, a computing device 704. The computing device704 may comprise an integrated display 706 that is configured to displayone or more user interface screens of the diagnostic application. Thecomputing device 704 may be communicatively coupled to an ultrasounddevice (not shown) using a wired or wireless communication link.

FIG. 7A shows an example home screen that may be displayed upon thediagnostic application being launched. Information that may be presentedon the home screen include an application title 702, an applicationdescription 703, and a sponsor region 708. The sponsor region 708 maydisplay information, for example, indicating the name, symbol, or logoof any sponsoring entity providing the diagnostic application. In thecase of a heart failure diagnostic application, a pharmaceuticalmanufacturer that provides one or more medications or therapies fortreating such a condition may sponsor the application. The home screenmay further include a selection region that allows the operator toperform various functions within the diagnostic application such as:schedule a follow-up examination with the subject, access more medicalresources, or begin a new diagnosis.

The computing device 704 may transition from the home screen to aclinical Q&A screen shown in FIG. 7B responsive to the “Begin NewDiagnosis” button in selection region 710 being activated in the homescreen shown in FIG. 7A. The clinical Q&A screen may pose one or moreclinical questions 712 to the operator. For a heart failure diagnosisapplication, an appropriate clinical question 712 posed to the operatormay be: “Is the patient experiencing paroxysmal nocturnal dyspnea?”Paroxysmal nocturnal dyspnea may be attacks of severe shortness ofbreath and coughing that generally occur at night. Such attacks may be asymptom of congestive heart failure. The diagnostic application mayreceive an answer to the clinical question in the response region 712.As will also be noted from FIG. 7B, the sponsor region 708 may continueto be provided in the diagnostic application. The sponsor region 708 maycomprise a link to exit the diagnostic application to a site hosted bythe sponsor.

The computing device 704 may transition from the clinical Q&A screen toan examination screen responsive to the “Yes” button being activated inresponse region 714. The examination screen may pose one or moreexamination questions 718 to the operator. For a heart failurediagnostic application, the examination question 718 may be to determinea current heart rate of the subject to be diagnosed. The diagnosticapplication may receive a response through the response region 720. Forexample, the operator may indicate that the heart rate of the subject isbelow a first value (e.g., less than 91 beats per minute (bpm)), withina range between the first value and a second value (e.g., between 91 and110 bpm), or above the second value (e.g., more than 110 bpm) in theresponse region 720.

Once the computing device 704 has received a response to the examinationquestion 718, the computing device 704 may transition from theexamination screen shown in FIG. 7C to an ultrasound image acquisitionscreen shown in FIG. 7D. The ultrasound image acquisition screen maypresent an imaging instruction 722 to the operator. For a heart failurediagnostic application, the imaging instruction 722 may instruct theoperator to begin an assisted ejection fraction (EF) measurement of thesubject. EF may be a measure of how much blood a heart ventricle pumpsout with each contraction. The EF may be identified be computed by, forexample, analyzing one or more ultrasound images of a heart of thesubject. The computing device 704 may begin an assisted EF measurementprocess responsive to the “Begin Measurement” button in selection region724 being activated.

The computing device 702 may communicate with an ultrasound device tocapture ultrasound images response to the “Begin Measurement” buttonbeing activated in the selection region 724. The computing device 702may also transition from the image acquisition screen shown in FIG. 7Dto an image acquisition assistance screen shown in FIG. 7E. The imageacquisition assistance screen may display an ultrasound image 726captured using the ultrasound device. In some embodiments, the imageacquisition assistance screen may display one or more instructionsregarding how to reposition the ultrasound device to obtain anultrasound image that contains the target anatomical view (e.g., a PLAXview). Once the ultrasound device has been properly positioned, theimage acquisition assistance screen may display an indication that theultrasound device is properly positioned. When a suitable (clinicallyrelevant) image(s) is obtained, the operator may confirm the acquisitionvia the “Confirm” button.

The computing device 704 may transition from the image acquisitionassistance screen shown in FIG. 7E to a diagnostic results screen shownin FIG. 7F once the ultrasound images have been confirmed by theoperator. The diagnostic results screen may display diagnostic results728, 732 determined from analyzing the captured ultrasound image 730. Asshown, the diagnostic results screen may display an EF of 30% for thesubject and an associated New York Heart Association (NYHA)classification of IV. This classification system utilizes fourcategories of heart failure, from I-IV with IV being the most severe.

The computing device 704 may transition from the diagnostic resultsscreen shown in FIG. 7F to one or more of the treatment screens shown inFIGS. 7G and 7H responsive to the “view possible treatments” buttonbeing activated in selection region 734. The treatment screen shown inFIG. 7G may display a treatment question 736 regarding a currenttreatment being provided to the subject and suggested treatments 738determined based on, for example, any one of the following: (1) aresponse to the treatment question 736, (2) diagnostic results, (3) thecaptured ultrasound image, (4) a response to the physical examinationquestion, and/or (5) a response to the clinical question. The treatmentscreen shown in FIG. 7H may be an extension of the treatment screen inFIG. 7G. For example, an operator may access the treatment screen inFIG. 7H by scrolling down from the treatment screen shown in FIG. 7G.The treatment screen in FIG. 7H may display a treatment selection 740where an operator may select which treatment they want to provide to thesubject. As shown, the treatment selection 740 may allow an operator topick between one or more medications to treat heart failure such asangiotensin-converting-enzyme inhibitors (ACE inhibitors), angiotensinreceptor blockers (ARB), or other alternatives. The diagnosticapplication may, then, display one or more external links 742 based onthe selected treatment to provide more information to the operatorregarding the treatment.

It should be appreciated that diagnostic application shown in FIGS.7A-7H is only one example implementation and other diagnosticapplications may be created for other conditions separate and apart fromcongestive heart failure. Further, diagnostic applications may becreated for use by a subject at-home (instead of a physician). Forexample, a physician may issue an ultrasound device configured forin-home use by a subject to monitor a condition of the subject using theultrasound device. A diagnostic application may also be provided to thesubject to use with the ultrasound device. Such a diagnostic applicationmay be installed on a personal mobile smartphone or tablet of thesubject. The diagnostic application may be configured to assist thesubject to operate the ultrasound device and store (and/or upload) thecaptured ultrasound images for analysis by the physician. Thereby, thephysician may be able to remotely monitor a condition of the subjectwithout making the subject remain in inpatient care.

FIGS. 8A-8D show an example user interface for such a diagnosticapplication that is designed to be used by a subject in an at-homeenvironment. The diagnostic application may be configured to assist anoperator (e.g., the subject) use an ultrasound device to captureultrasound images in an at-home setting. The diagnostic application maybe executed by, for example, a computing device 804 (such as a mobilesmartphone or a tablet of the subject). The computing device 804 maycomprise an integrated display 806 that is configured to display one ormore user interface screens of the diagnostic application. The computingdevice 804 may be communicatively coupled to an ultrasound device (notshown) using a wired or wireless communication link. The computingdevice may also comprise an imaging device 805 (e.g., a camera) that isconfigured to capture non-acoustic images. The imaging device 805 may bedisposed on a same side as the display 806 to allow the operator tosimultaneously capture images of themselves holding an ultrasound devicewhile viewing one or more instructions displayed on the display 806.

FIG. 8A shows an example home screen that may be displayed upon thediagnostic application being launched. The home screen includes amessage 808 to the operator to instruct the operator to scan a quickresponse (QR) code associated with the ultrasound device. The QR codemay be, for example, disposed on the ultrasound device itself and/ordisposed on a packaging associated with the ultrasound device. The homescreen may also display images captured by an imaging device (e.g.,integrated into the computing device 804 and disposed on a side oppositethe display 806). The home screen may show a scanning region 810 in thecaptured images to illustrate where a user should place the QR code inthe field of view of the imaging device to have the QR code read.

Once the computing device 804 reads the QR code, the computing device804 may transition from the home screen shown in FIG. 8A to a subjectinformation screen shown in FIG. 8B. The subject information screen mayinclude a display of subject information 810 obtained by the computingdevice 804 using the scanned QR code. For example, the computing device804 may have employed the scanned QR code to access medical records ofthe subject in a remote server. Once the operator has confirmed that thesubject information 810 is correct, the operator may activate theconfirm button in the selection region 812.

It should be appreciated that other types of bar codes may be employedseparate from QR codes. Other example bar codes include: MaxiCode barcodes, Codabar bar codes, and Aztec bar codes.

The computing device 804 may transition from the subject informationscreen shown in FIG. 8B to the image acquisition screen shown in FIG. 8Cresponsive to the “Confirm” button being activated in the selectionregion 812. As shown, the image acquisition screen includes a message814 for the operator to apply gel to the ultrasound device and aselection region 816 including a being button for the operator to beginacquisition of ultrasound images.

The computing device 804 may transition from the image acquisitionscreen shown in FIG. 8C to the image acquisition assistance screen shownin FIG. 8D responsive to the “Begin” button being activated in theselection region 816. As shown, the image acquisition assistance screenmay include a non-acoustic image (e.g., captured by the imaging device805) of a subject 818 holding an ultrasound device 820. An instruction822 may be superimposed over the captured non-acoustic image to guidethe operator (e.g., the subject) to capture an ultrasound imagecontaining a target anatomical view. Once the computing device 804 hascaptured the ultrasound image containing the target anatomical view, thecomputing device 804 may locally store the captured ultrasound image forlater retrieval by a physician and/or upload the image to an externalserver to be added to a set of medical records associated with thesubject. The computing device 804 may further display a confirmation tothe operator that the ultrasound image was successfully captured.

Example Processes

FIG. 9 shows an example process 900 for guiding an operator of anultrasound device to capture an ultrasound image that contains a targetanatomical view. The process 900 may be performed by, for example, acomputing device in an ultrasound system. As shown, the process 900comprises an act 902 of obtaining an ultrasound image, an act 904 ofdetermining whether the ultrasound image contains the target anatomicalview, an act 906 of generating a guidance plan, an act 908 of providinginstructions to reposition the ultrasound device, and an act 910 ofproviding an indication of proper positioning.

In act 902, the computing device may obtain an ultrasound image of thesubject. The computing device may obtain the ultrasound image bycommunicating with an ultrasound device communicatively coupled to thecomputing device. For example, the computing device may send aninstruction to the ultrasound device to generate ultrasound data andsend the ultrasound data to the computing device. The computing devicemay, in turn, use the received ultrasound data to generate theultrasound image. Additionally (or alternatively), the ultrasound imagemay be generated by the ultrasound device and the computing device mayretrieve the ultrasound image from the ultrasound device.

In act 904, the computing device may determine whether the ultrasoundimage contains the target anatomical view. If the computing devicedetermines that the ultrasound image contains the target anatomicalview, the computing device may proceed act 910 and provide an indicationof proper positioning. Otherwise the system may proceed to act 906 togenerate a guidance plan for the operator to move the ultrasound device.

In some embodiments, the computing device may employ an automated imageprocessing technique, such as a deep learning technique, to determinewhether the ultrasound image contains the target anatomical view. Forexample, the ultrasound image may be provided as input to a neuralnetwork that is trained to identify an anatomical view contained in theultrasound image. The output of such a neural network may be anindication of the particular anatomical view that is contained in theultrasound image. In this example, the identified anatomical view may becompared with the target anatomical view to determine whether theymatch. If the identified anatomical view and the target anatomical viewmatch, the computing device may determine that the ultrasound imagecontains the target anatomical view. Otherwise, the computing device maydetermine that the ultrasound image does not contain the anatomicalview. In another example, the neural network may be configured todirectly provide an indication of an instruction for the operator basedon an input ultrasound image. Thereby, the neural network may provide asan output a confirmation that the ultrasound devices properly positionedor an instruction to move the ultrasound device in a particulardirection. In this example, the computing device may determine that theultrasound image contains the target anatomical view responsive to theneural network providing a confirmation as an output. Otherwise, thecomputing device may determine that the ultrasound image does notcontain the anatomical view.

In act 906, the computing device may generate a guidance plan regardinghow to guide the operator to move the ultrasound device. In someembodiments, the guidance plan may comprise a guide path along which theoperator should move the ultrasound device from an initial position to atarget position where an ultrasound image containing the targetanatomical view may be captured. In these embodiments, the computingdevice may identify the initial position of the ultrasound device on thesubject at least in part by: identifying an anatomical view contained inthe ultrasound image (e.g., using deep learning techniques) and map theidentified anatomical view to a position on the subject. The targetposition may be identified by, for example, mapping the targetanatomical view to a position on the subject. Once the initial andtarget positions have been identified, the computing device may identifya guide path between the initial and target positions along which theultrasound device should move. The guide path may comprise a sequence ofdirections (e.g., translational directions or rotational directions) forthe ultrasound device to travel along to reach the target position. Thegenerated guide path may not be the shortest path between the initialposition of the ultrasound device in the target position of theultrasound device. For example, the generated path may avoid usingdiagonal movements that may be challenging to properly convey to theoperator. Alternatively (or additionally), the generated path may avoidcertain areas of the subject such as areas comprising hard tissues. Oncethe guide path between the initial position and the target position ofthe ultrasound device has been determined, the computing device maygenerate a sequence of one or more instructions to provide to theoperator to instruct the operator to move the ultrasound device alongthe guide path.

In act 908, the computing device may provide an instruction toreposition the ultrasound device to the operator. The instruction maybe, for example, an audible instruction played through a speaker, avisual instruction displayed using a display, and/or a tactileinstruction provided using a vibration device (e.g., integrated into thecomputing device and/or the ultrasound device). The instruction may beprovided based on, for example, the sequence of instructions in theguidance plan generated in act 906. For example, the computing devicemay identify a single instruction from the sequence of instructions andprovide the identified instruction. It should be appreciated that theinstruction need not originate from a guidance plan. For example, asdiscussed above, a neural network may be configured to directly outputan instruction based on a received ultrasound image. In this example,the output instruction may be directly provided and the act 906 ofgenerating a guidance plan may be omitted.

Once the computing device has provided the instruction to reposition theultrasound device, the computing device may repeat one or more of acts902, 904, 906 and/or 908 to provide the operator additionalinstructions.

In act 910, the computing device may provide an indication of properpositioning. For example, the computing device may provide an audibleconfirmation played through a speaker, a visual confirmation displayedusing a display, or a tactile confirmation provided through a vibrationdevice.

FIG. 10 shows an example process 1000 for providing an augmented realityinterface for an operator. The augmented reality interface may include anon-acoustic image of a real-world environment including an ultrasounddevice and one or more elements (such as instructions) overlaid onto thenon-acoustic image. The process 1000 may be performed by, for example, acomputing device in an ultrasound system. As shown in FIG. 10, theprocess 1000 comprises an act 1002 of obtaining an image of anultrasound device, an act 1003 of generating a composite image, and anact 1008 of presenting the composite image. The act 1003 of generatingthe composite image may comprise an act 1004 of identifying a pose of anultrasound device in the image and an act 1006 of overlaying theinstruction onto the image using the identified pose.

In act 1002, the computing device may capture an image (e.g., anon-acoustic image) of the ultrasound device. The non-acoustic image maybe captured by an imaging device (e.g., a camera) integrated into thecomputing device. For example, the non-acoustic image may be capturedusing a front-facing camera of a mobile smartphone (e.g., on the sameside as the display) when the operator is also the subject. In anotherexample, the non-acoustic image may be captured using a rear-facingcamera of a mobile smartphone (e.g., on the opposite side as thedisplay) when the operator is a person (or group of people) separatefrom the subject.

In act 1003, the computing may generate the composite image. Thecomposite image may comprise the non-acoustic image captured in act 1002and one or more elements overlaid onto the non-acoustic image. The oneor more elements overlaid onto the non-acoustic image may be, forexample, one or more instructions designed to provide feedback to theoperator regarding how to reposition the ultrasound device to obtain anultrasound image that contains a target anatomical view. The computingdevice may generate the composite image in any of a variety of ways. Insome embodiments, the computing device may be configured to generate thecomposite image by performing acts 1004 and 1006.

In act 1004, the computing device may identify a pose (e.g., positionand/or orientation) of the ultrasound device in the non-acoustic image.The computing device may identify the pose of the ultrasound device inthe captured image using an automated image processing technique (e.g.,a deep learning technique). For example, the non-acoustic image may beprovided as an input to a neural network that is configured to identifywhich pixels in the non-acoustic image are associated with theultrasound device. In this example, the computing device may use theidentified pixels to determine a position of the ultrasound device inthe non-acoustic image. In some embodiments, the ultrasound device mayhave a marker disposed thereon that is visible in the image to easeidentification of the ultrasound device in the non-acoustic image. Themarker may have a distinct shape, color, and/or image that is easy torecognize using an automated image processing technique. Additionalinformation may also be employed to identify the pose of the ultrasounddevice in combination with (or in place of) the information extractedfrom the non-acoustic image. For example, the ultrasound device maycomprise one or more sensors configured to detect movement (e.g.,accelerometers, gyroscopes, compasses, and/or inertial measurementunits). In this example, movement information from these sensors in theultrasound device may be employed to determine the pose of theultrasound device. In another example, the ultrasound device (e.g.,ultrasound device 502) and a computing device (e.g., computing device504) connected to the ultrasound device may comprise sensors configuredto detect movement. In this example, the movement information from thesensors in both the ultrasound device and the computing device may beused in concert to identify the pose of the ultrasound device relativeto the computing device and, thereby, identify the pose of theultrasound device in the captured non-acoustic image.

In act 1006, the computing device may overlay an instruction onto thenon-acoustic image using the identified pose to form an augmentedreality interface. For example, the computing device may overlay aninstruction regarding how to move the ultrasound device (e.g., adirectional arrow) onto the non-acoustic image so as to be proximateand/or partially covering the ultrasound device. Additionally (oralternatively), the pose may be employed to position other elements inthe augmented reality interface. For example, the pose of the ultrasounddevice may be employed to position an ultrasound image in the augmentedreality interface. In this example, the ultrasound image may bepositioned in the augmented reality interface so as to appear to beextending from the ultrasound device in the non-acoustic image into thesubject. Thereby, the operator may gain an appreciation for theparticular portion of the subject that is being imaged with theultrasound device.

In act 1008, the computing device may present a composite image to theoperator. For example, the computing device may present the compositeimage to the operator using a display integrated into the computingdevice. Alternative (or additionally), the computing device may transmitthe composite image to another device (e.g., to be presented on adisplay of the other device).

FIG. 11 shows an example process 1100 for tracking the location of anultrasound device in non-acoustic images using a marker disposedthereon. The process 1100 may be performed by, for example, a computingdevice in an ultrasound system. As shown, the process 1100 comprises anact 1102 of obtaining an image of a marker disposed on an ultrasounddevice, an act 1103 of identifying a pose of the ultrasound device, andan act 1108 of presenting an instruction using the identified pose. Theact 1103 of identifying the pose of the ultrasound device may comprisean act 1104 of identifying the location of the marker in the image andan act 1106 of analyzing a characteristic of the marker.

In act 1102, the computing device may capture a non-acoustic image ofthe marker on the ultrasound device. The non-acoustic image may becaptured by an imaging device (e.g., a camera) integrated into thecomputing device. For example, the non-acoustic image may be capturedusing a front-facing camera of a mobile smartphone (e.g., on the sameside as the display) when the operator is also the subject. In anotherexample, the non-acoustic image may be captured using a rear-facingcamera of a mobile smartphone (e.g., on the opposite side as thedisplay) when the operator is person (or group of people) separate fromthe subject.

In act 1103, the computing device may identify a pose (e.g., a positionand/or orientation) of the ultrasound device in the captured image usingthe marker. The computing device may identify the pose of the ultrasounddevice in the captured image in any of a variety of ways. In someembodiments, the computing device may identify the pose of theultrasound device in the non-acoustic image by performing acts 1104 and1106.

In act 1104, the computing device may identify a location of the markerin the non-acoustic image. The computing device may use the identifiedlocation of the marker to identify a position of the ultrasound deviceon which the marker is disposed. The location of the marker may bedetermined by, for example, locating one or more features characteristicto the marker, such as a shape, color, and/or image, in the image usingan automated image processing technique.

In act 1106, the computing device may analyze a characteristic of themarker. The computing device may analyze a characteristic of the markerto, for example, determine an orientation of the ultrasound device inthe captured image. The particular way in which the computing devicedetermines the orientation using characteristics of the marker maydepend on, for example, the particular marker employed. In one example,the marker may be a monochrome marker comprising a pattern. In thisexample, the pattern may be analyzed in order to determine inorientation of the pattern and, thereby, determine an orientation of theultrasound device in the non-acoustic image. In another example, themarker may be a dispersive marker that is configured to presentdifferent colors depending on the viewing angle. In this example, thecomputing device may identify a color of the marker in the non-acousticimage and use the identified color to determine an orientation of themarker and, thereby, an orientation of the ultrasound device. In yetanother example, the marker may be a holographic marker that isconfigured to present different images depending on the viewing angle.In this example, the computing device may identify an image presented bythe holographic marker and use the identified image to determine anorientation of the marker, and thereby, an orientation of the ultrasounddevice.

In act 1108, the computing device may present an instruction using theidentified pose. In some embodiments, the computing device may overlaythe instruction onto the non-acoustic image obtained in act 1102 usingthe identified pose to form a composite image for an augmented realityinterface. For example, the computing device may overlay an instructionregarding how to move the ultrasound device (e.g., a directional arrow)onto the non-acoustic image so as to be proximate and/or partiallycovering the ultrasound device. Additionally (or alternatively), thepose may be employed to position other elements in the augmented realityinterface. For example, the pose of the ultrasound device may beemployed to position an ultrasound image in the augmented realityinterface. In this example, the ultrasound image may be positioned inthe augmented reality interface so as to appear to be extending from theultrasound device in the non-acoustic image into the subject. Thereby,the operator may gain an appreciation for the particular portion of thesubject that is being imaged with the ultrasound device.

FIG. 12 shows an example process 1200 for analyzing captured ultrasoundimages to identify a medical parameter of the subject. The process 1200may be performed by, for example, a computing device in an ultrasoundsystem. As shown, the process 1200 comprises an act 1202 of obtaining anultrasound image, an act 1204 of identifying an anatomical feature inthe ultrasound image, and an act 1206 of identifying a medical parameterusing the identified anatomical feature.

In act 1202, the computing device may obtain an ultrasound image of thesubject. The computing device may obtain the ultrasound image bycommunicating with an ultrasound device communicatively coupled to thecomputing device. For example, the computing device may send aninstruction to the ultrasound device to generate ultrasound data andsend the ultrasound data to the computing device. The computing devicemay, in turn, use the received ultrasound data to generate theultrasound image.

In act 1204, the computing device may identify an anatomical feature inthe ultrasound image. For example, the computing device may identify aheart ventricle, a heart valve, a heart septum, a heart papillarymuscle, a heart atrium, an aorta, or a lung as an anatomical feature inthe ultrasound image. The computing device may identify the anatomicalfeature using an automated image processing technique, such as a deeplearning technique. For example, the computing device may provide theultrasound image as an input to a neural network that is configured(e.g., trained) to provide, as an output, an indication of which pixelsin the ultrasound image are associated with an anatomical feature. Itshould be appreciated that this neural network may be separate anddistinct from any neural networks employed to guide an operator toobtain an ultrasound image containing a target anatomical view (such asthose employed in process 900 described above).

In act 1206, the computing device may identify a medical parameter usingthe identified anatomical feature. For example, the computing device maydetermine an ejection fraction, a fractional shortening, a ventriclediameter, a ventricle volume, an end-diastolic volume, an end-systolicvolume, a cardiac output, a stroke volume, an intraventricular septumthickness, a ventricle wall thickness, or a pulse rate of the subject.In some embodiments, the computing device may identify the medicalparameters by analyzing one or more characteristics of the identifiedanatomical feature. For example, the computing device may identify aheart ventricle in the ultrasound image and the dimensions of the heartventricle may be extracted from the ultrasound image to determine aventricle volume and/or a ventricle diameter. It should be appreciatedthat the computing device may analyze more than a single ultrasoundimage to identify the medical parameter. For example, the computingdevice may identify a ventricle volume in each of a plurality ofultrasound images and select a lowest ventricle volume as theend-systolic volume and select the highest ventricle volume as theend-diastolic volume. Further, the end-systolic and end-diastolicvolumes may be employed to determine another medical parameter, such asan EF.

FIG. 13 shows an example process 1300 for generating a diagnosis of amedical condition of a subject. The process 1300 may be performed by,for example, a computing device in an ultrasound system. As shown, theprocess 1300 comprises an act 1302 of receiving medical informationabout the subject, an act 1304 of identifying a target anatomical view,an act 1306 of obtaining an ultrasound image containing the targetanatomical view, an act 1308 of generating a diagnosis of a medicalcondition of a subject, and an act 1310 of generating recommendedtreatments for the subject.

In act 1302, the computing device may receive medical information aboutthe subject. Example medical information about the subject that may bereceived includes: a heart rate, a blood pressure, a body surface area,an age, a weight, a height, and a medication being taken by the subject.The computing device may receive the medical information by, forexample, posing one or more questions to an operator and receiving aresponse. Additionally (or alternatively), the computing device maycommunicate with an external system to obtain the medical information.For example, the operator may scan a barcode (e.g., a QR code) on theultrasound device using the computing device and the computing devicemay use information obtained from the barcode to access medical recordsassociated with the subject on a remote server.

In act 1304, the computing device may identify a target anatomical viewbased on the received medical information. The computing device mayanalyze the received medical information to identify one or more organsthat may be functioning abnormally. Then, the computing device mayidentify an anatomical view that contains the identified one or moreorgans. For example, the medical information about the subject mayindicate that the heart of the subject is functioning abnormally (e.g.,the patient has symptoms of congestive heart failure) and identify aPLAX view as the anatomical view to image.

In act 1306, the computing device may obtain an ultrasound imagecontaining the target anatomical view. For example, the computing devicemay retrieve an ultrasound image of the subject containing the targetanatomical view from an electronic health record of the patient.Alternatively (or additionally), the computing device may guide theoperator to obtain an ultrasound image that contains the targetanatomical view. For example, the computing device may issue one or moreinstructions regarding how the operator should position the ultrasounddevice on the subject to obtain an ultrasound image containing thetarget anatomical view. The computing device may generate and/or providethese instructions in any of a variety of ways. For example, thecomputing device may perform a process that is similar to (or identicalto) the process 900 described above.

In act 1308, the computing device may generate a diagnosis of a medicalcondition of the subject using the ultrasound image containing thetarget anatomical view. In some embodiments, the computing device mayanalyze the ultrasound image containing the target anatomical view toidentify one or more medical parameters (e.g., an EF of the subject) anduse the identified one or more medical parameters (alone or incombination with other information such as medical information of thesubject) to generate the diagnosis. In these embodiments, the computingdevice may perform one or more acts in process 1200 to identify amedical parameter of the subject. For example, the computing device maydetermine an ejection fraction of the subject by performing acts 1202,1204, and/or 1206 and compare the resulting ejection fraction value witha threshold to determine whether the subject is likely suffering fromcongestive heart failure. The computing device may combine theinformation regarding the medical parameters with other information(such as the medical information about the subject received in act 1302)to diagnose a medical condition of the subject. For example, thecomputing device may diagnose a patient with congestive heart failureresponsive to the computing device determining that the ejectionfraction of the subject is below a threshold and that the subject hasreported symptoms of congestive heart failure (such as experiencingparoxysmal nocturnal dyspnea). It should be appreciated that thecomputing device may be configured to diagnose any of a variety ofmedical conditions such as: heart conditions (e.g., congestive heartfailure, coronary artery disease, and congenital heart disease), lungconditions (e.g., lung cancer), kidney conditions (e.g., kidney stones),and/or joint conditions (e.g., arthritis).

In act 1310, the computing device may generate one or more recommendedtreatments for the subject. The computing device may generate the one ormore recommended treatments based on the diagnosis of the subject.Example recommended treatments include: a change in diet, a change inexercise routine, a pharmaceutical drug, a biologic (e.g., vaccines,gene therapies, cellular therapies), radiotherapy, chemotherapy, andsurgical intervention. For example, the subject may be diagnosed withcongestive heart failure and the computing device generate a recommendedtreatment of: angiotensin-converting-enzyme inhibitors (ACE inhibitors),angiotensin receptor blockers (ARB), or other alternatives.

It should be appreciated that the computing device may use informationother than the diagnosis to generate the recommended treatment, such asmedical information of the subject and/or one or more medical parametersextracted from the ultrasound image. For example, the medicalinformation of the subject may indicate that the subject is a smoker andthe computing device may include a recommended treatment of quittingsmoking when the subject is diagnosed with congestive heart failure. Inanother example, the medical information of the subject may include oneor more drug allergies of the subject and the computing device may notrecommend any treatments that involve administration of a drug to whichthe subject is allergic. In yet another example, the medical informationof the subject may include one or more drugs taken by the subject andthe computing device may not recommend any treatments that wouldadversely interact with one or more of the drugs already taken by thesubject.

Various inventive concepts may be embodied as one or more processes, ofwhich examples have been provided. The acts performed as part of eachprocess may be ordered in any suitable way. Thus, embodiments may beconstructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments. Further,one or more of the processes may be combined. For example, the process1300 of identifying an anatomical view to image based on medicalinformation about the subject may be combined with the process 1200 foranalyzing captured ultrasound images to identify a medical parameter ofthe subject. Thereby, the computing device may (1) identify ananatomical view to image, (2) guide the operator to capture anultrasound image containing the anatomical view, and (3) analyze thecaptured ultrasound image to identify medical parameters of the subject.In this example, the ultrasound device may additionally make one or moretreatment recommendations based on the identified medical parametersand/or medical information regarding the subject.

Example Deep Learning Techniques

Aspects of the technology described herein relate to the application ofautomated image processing techniques to analyze images, such asultrasound images and non-acoustic images. In some embodiments, theautomated image processing techniques may comprise machine learningtechniques such as deep learning techniques. Machine learning techniquesmay comprise techniques that seek to identify patterns in a set of datapoints and use the identified patterns to make predictions for new datapoints. These machine learning techniques may involve training (and/orbuilding) a model using a training data set to make such predictions.The trained model may be used as, for example, a classifier that isconfigured to receive a data point as an input and provide an indicationof a class to which the data point likely belongs as an output.

Deep learning techniques may include those machine learning techniquesthat employ neural networks to make predictions. Neural networkstypically comprise a collection of neural units (referred to as neurons)that each may be configured to receive one or more inputs and provide anoutput that is a function of the input. For example, the neuron may sumthe inputs and apply a transfer function (sometimes referred to as an“activation function”) to the summed inputs to generate the output. Theneuron may apply a weight to each input to, for example, weight someinputs higher than others. Example transfer functions that may beemployed include step functions, piecewise linear functions, and sigmoidfunctions. These neurons may be organized into a plurality of sequentiallayers that each comprise one or more neurons. The plurality ofsequential layers may include an input layer that receives the inputdata for the neural network, an output layer that provides the outputdata for the neural network, and one or more hidden layers connectedbetween the input and output layers. Each neuron in a hidden layer mayreceive inputs from one or more neurons in a previous layer (such as theinput layer) and provide an output to one or more neurons in asubsequent layer (such as an output layer).

A neural network may be trained using, for example, labeled trainingdata. The labeled training data may comprise a set of example inputs andan answer associated with each input. For example, the training data maycomprise a plurality of ultrasound images that are each labeled with ananatomical view that is contained in the respective ultrasound image. Inthis example, the ultrasound images may be provided to the neuralnetwork to obtain outputs that may be compared with the labelsassociated with each of the ultrasound images. One or morecharacteristics of the neural network (such as the interconnectionsbetween neurons (referred to as edges) in different layers and/or theweights associated with the edges) may be adjusted until the neuralnetwork correctly classifies most (or all) of the input images.

In some embodiments, the labeled training data may comprise samplepatient images are obtained that need not all be “standard” or “good”image of an anatomic structure. For example, one or more of the samplepatient images may be “non-ideal” for training purposes. Each of thesesample patient images may be evaluated by a trained clinician. Thetrained clinician may add a qualitative label to each of the samplepatient images. In the specific example of a PLAX image, the clinicianmay determine that the given image is “normal” (i.e., depicts a goodview of the structure for analysis purposes). In the alternative, if theimage is not ideal, the clinician may provide a specific label for theimage that describes the problem with it. For example, the image mayrepresent an image taken because the ultrasound device was oriented “toocounterclockwise” or perhaps “too clockwise” on the patient. Any numberof specific errors may be assigned to a sample given image.

Once the training data has been created, the training data may be loadedto a database (e.g., an image database) and used to train a neuralnetwork using deep learning techniques. Once the neural network has beentrained, the trained neural network may be deployed to one or morecomputing devices. It should be appreciated that the neural network maybe trained with any number of sample patient images. For example, aneural network may be trained with as few as 7 or so sample patientimages, although it will be appreciated that the more sample imagesused, the more robust the trained model data may be.

Convolutional Neural Networks

In some applications, a neural network may implemented using one or moreconvolution layers to form a convolutional neural network. An exampleconvolutional neural network is shown in FIG. 14 that is configured toanalyze an image 1402. As shown, the convolutional neural networkcomprises an input layer 1404 to receive the image 1402, an output layer1408 to provide the output, and a plurality of hidden layers 1406connected between the input layer 1404 and the output layer 1408. Theplurality of hidden layers 1406 comprises convolution and pooling layers1410 and dense layers 1412.

The input layer 1404 may receive the input to the convolutional neuralnetwork. As shown in FIG. 14, the input the convolutional neural networkmay be the image 1402. The image 1402 may be, for example, an ultrasoundimage or a non-acoustic image.

The input layer 1404 may be followed by one or more convolution andpooling layers 1410. A convolutional layer may comprise a set of filtersthat are spatially smaller (e.g., have a smaller width and/or height)than the input to the convolutional layer (e.g., the image 1402). Eachof the filters may be convolved with the input to the convolutionallayer to produce an activation map (e.g., a 2-dimensional activationmap) indicative of the responses of that filter at every spatialposition. The convolutional layer may be followed by a pooling layerthat down-samples the output of a convolutional layer to reduce itsdimensions. The pooling layer may use any of a variety of poolingtechniques such as max pooling and/or global average pooling. In someembodiments, the down-sampling may be performed by the convolution layeritself (e.g., without a pooling layer) using striding.

The convolution and pooling layers 1410 may be followed by dense layers1412. The dense layers 1412 may comprise one or more layers each withone or more neurons that receives an input from a previous layer (e.g.,a convolutional or pooling layer) and provides an output to a subsequentlayer (e.g., the output layer 1408). The dense layers 1412 may bedescribed as “dense” because each of the neurons in a given layer mayreceive an input from each neuron in a previous layer and provide anoutput to each neuron in a subsequent layer. The dense layers 1412 maybe followed by an output layer 1408 that provides the output of theconvolutional neural network. The output may be, for example, anindication of which class, from a set of classes, the image 1402 (or anyportion of the image 1402) belongs to.

It should be appreciated that the convolutional neural network shown inFIG. 14 is only one example implementation and that otherimplementations may be employed. For example, one or more layers may beadded to or removed from the convolutional neural network shown in FIG.14. Additional example layers that may be added to the convolutionalneural network include: a rectified linear units (ReLU) layer, a padlayer, a concatenate layer, and an upscale layer. An upscale layer maybe configured to upsample the input to the layer. An ReLU layer may beconfigured to apply a rectifier (sometimes referred to as a rampfunction) as a transfer function to the input. A pad layer may beconfigured to change the size of the input to the layer by padding oneor more dimensions of the input. A concatenate layer may be configuredto combine multiple inputs (e.g., combine inputs from multiple layers)into a single output.

Convolutional neural networks may be employed to perform any of avariety of functions described herein. For example, a convolutionalneural networks may be employed to: (1) identify an anatomical viewcontained in an ultrasound image, (2) identify an instruction to providean operator, (3) identify an anatomical feature in an ultrasound image,or (4) identify a pose of ultrasound device in a non-acoustic image. Itshould be appreciated that more than a single convolutional neuralnetwork may be employed to perform one or more of these functions. Forexample, a first convolutional neural network may be employed toidentify an instruction to provide an operator based on an inputultrasound image and a second, different convolutional neural networkmay be employed to identify an anatomical feature in an ultrasoundimage. The first and second neural networks may comprise a differentarrangement of layers and/or be trained using different training data.

An example implementation of a convolutional network is shown below inTable 1. The convolutional neural network shown in Table 1 may beemployed to classify an input image (e.g., an ultrasound image). Forexample, the convolutional network shown in Table 1 may be configured toreceive an input ultrasound image and provide an output that isindicative of which instruction from a set of instructions should beprovided to an operator to properly position the ultrasound device. Theset of instructions may include: (1) tilt the ultrasound deviceinferomedially, (2) rotate the ultrasound device counterclockwise, (3)rotate the ultrasound device clockwise, (4) move the ultrasound deviceone intercostal space down, (5) move the ultrasound device oneintercostal space up, and (6) slide the ultrasound device medially. InTable 1, the sequence of the layer is denoted by the “Layer Number”column, the type of the layer is denoted by the “Layer Type” column, andthe input to the layer is denoted by the “Input to Layer” column.

TABLE 1 Example Layer Configuration for Convolutional neural networkLayer Number Layer Type Input to Layer 1 Input Layer Input Image 2Convolution Layer Output of Layer 1 3 Convolution Layer Output of Layer2 4 Pooling Layer Output of Layer 3 5 Convolution Layer Output of Layer4 6 Convolution Layer Output of Layer 5 7 Pooling Layer Output of Layer6 8 Convolution Layer Output of Layer 7 9 Convolution Layer Output ofLayer 8 10 Pooling Layer Output of Layer 9 11 Convolution Layer Outputof Layer 10 12 Convolution Layer Output of Layer 11 13 Pooling LayerOutput of Layer 12 14 Fully Connected Layer Output of Layer 13 15 FullyConnected Layer Output of Layer 14 16 Fully Connected Layer Output ofLayer 15

Another example implementation of a convolutional neural network isshown below in Table 2. The convolutional neural network in Table 2 maybe employed to identify two points on the basal segments of the leftventricle in an ultrasound image. In Table 2, the sequence of the layeris denoted by the “Layer Number” column, the type of the layer isdenoted by the “Layer Type” column, and the input to the layer isdenoted by the “Input to Layer” column.

TABLE 2 Example Layer Configuration for Convolutional neural networkLayer Number Layer Type Input to Layer 1 Input Layer Input Image 2Convolution Layer Output of Layer 1 3 Convolution Layer Output of Layer2 4 Pooling Layer Output of Layer 3 5 Convolution Layer Output of Layer4 6 Convolution Layer Output of Layer 5 7 Pooling Layer Output of Layer6 8 Convolution Layer Output of Layer 7 9 Convolution Layer Output ofLayer 8 10 Pooling Layer Output of Layer 9 11 Convolution Layer Outputof Layer 10 12 Convolution Layer Output of Layer 11 13 Convolution LayerOutput of Layer 12 14 Fully Connected Layer Output of Layer 13 15 FullyConnected Layer Output of Layer 14 16 Fully Connected Layer Output ofLayer 15

Yet another example implementation of convolutional neural network isshown below in Table 3. The convolutional neural network shown in Table3 may be configured to receive an ultrasound image and classify eachpixel in the input image as belonging to the foreground (anatomicalstructure, e.g., left ventricle) or to the background. Relative to theconvolutional neural networks shown in Tables 1 and 2, upsampling layershave been introduced to increase the resolution of the classificationoutput. The output of the upsampled layers is combined with the outputof other layers to provide accurate classification of individual pixels.In Table 3, the sequence of the layer is denoted by the “Layer Number”column, the type of the layer is denoted by the “Layer Type” column, andthe input to the layer is denoted by the “Input to Layer” column.

TABLE 3 Example Layer Configuration for Convolutional neural networkLayer Number Layer Type Input to Layer 1 Input Layer Input Image 2Convolution Layer Output of Layer 1 3 Convolution Layer Output of Layer2 4 Pooling Layer Output of Layer 3 5 Convolution Layer Output of Layer4 6 Convolution Layer Output of Layer 5 7 Pooling Layer Output of Layer6 8 Convolution Layer Output of Layer 7 9 Convolution Layer Output ofLayer 8 10 Pooling Layer Output of Layer 9 11 Convolution Layer Outputof Layer 10 12 Convolution Layer Output of Layer 11 13 Convolution LayerOutput of Layer 12 14 Upscale Layer Output of Layer 13 15 ConvolutionLayer Output of Layer 14 16 Pad Layer Output of Layer 15 17 ConcatenateLayer Output of Layers 9 and 16 18 Convolution Layer Output of Layer 1719 Convolution Layer Output of Layer 18 20 Upscale Layer Output of Layer19 21 Convolution Layer Output of Layer 20 22 Pad Layer Output of Layer21 23 Concatenate Layer Output of Layers 6 and 22 24 Convolution LayerOutput of Layer 23 25 Convolution Layer Output of Layer 24 26 UpscaleLayer Output of Layer 25 27 Convolution Layer Output of Layer 26 28 PadLayer Output of Layer 27 29 Concatenate Layer Output of Layers 3 and 2830 Convolution Layer Output of Layer 29 31 Convolution Layer Output ofLayer 30 32 Convolution Layer Output of Layer 31

Integrating Statistical Knowledge into Convolutional Neural Networks

In some embodiments, statistical prior knowledge may be integrated intoa convolutional neural network. For example, prior statisticalknowledge, obtained through principal components analysis (PCA), may beintegrated into a convolutional neural network in order to obtain robustpredictions even when dealing with corrupted or noisy data. In theseembodiments, the network architecture may be trained end-to-end andinclude a specially designed layer which incorporates the dataset modesof variation discovered via PCA and produces predictions by linearlycombining them. Further, a mechanism may be included to focus theattention of the convolutional neural network on specific regions ofinterest of an input image in order to obtain refined predictions.

The complexity of anatomical structures along with the presence ofnoise, artifacts, visual clutter, and poorly defined image areas oftencause ambiguities and errors in image analysis. In the medical domain,many of these errors can be resolved by relying on statistical priorknowledge. For example, in segmentation it is useful to incorporateprior knowledge about the segmentation contour. Landmark localizationtasks can benefit from the semantic relationships between differentlandmarks and how their positions are allowed to change with respect toeach other. Finally, statistical models capturing the appearance ofselected regions have been shown to improve results in a number ofcases.

Shape models have also been used to constrain segmentation algorithmsthat are based on machine learning. This has been done by learning aposterior distribution of PCA coefficients and by re-projecting portionsof ground truth contours onto unseen examples. These models rely onshallow architectures, manually engineered or learned features and shapeconstraints being imposed as part of a regularization or post-processingstep.

Deep learning approaches and convolutional neural networks inparticular, have shown astonishing capabilities to learn a hierarchy offeatures directly from raw data. Deep learning models are organized inmultiple layers, where features are extracted in a cascaded fashion. Asthe depth of the network increases, the extracted features refer tobigger image regions and therefore recognize higher level conceptscompared to the ones extracted in earlier layers.

Unfortunately, the applicability of deep learning approaches in medicalimage analysis is often limited by the requirement to train with largeannotated datasets. Supplying more annotated data during the learningprocess allows a larger amount of challenging, real-world situations tobe captured and therefore partly overcomes the difficulty to integrateprior statistical knowledge in the learning process. In the medicaldomain, it is often difficult to obtain large annotated datasets due tolimitations on data usage and circulation and the tediousness of theannotation process. Moreover, medical images typically exhibit largevariability in the quality and appearance of the structures acrossdifferent scans, which further hampers the performances of machinevision algorithms. Ultrasound images, in particular, are often corruptedby noise, shadows, signal drop regions, and other artifacts that maketheir interpretation challenging even to human observers. Additionally,ultrasound scans exhibit high intra- and inter-operator acquisitionvariability, even when scanned by experts.

In some embodiments, PCA may be employed to advantageously discover theprincipal modes of variation of training data. Such discovered principlemodes of variation may be integrated into a convolutional neuralnetwork. The robustness of the results is increased by constraining thenetwork predictions with prior knowledge extracted by statisticallyanalyzing the training data. This approach makes it possible to processcases where the anatomy of interest appears only partially, itsappearance is not clear, or it visually differs from the observedtraining examples.

A convolutional network architecture may be employed that includes a newPCA layer that incorporates the dataset modes of variation and producespredictions as a linear combination of the modes. This process is usedin procedure that focuses the attention of the subsequent convolutionalneural network layers on the specific region of interest to obtainrefined predictions. Importantly, the network is trained end-to-end withthe shape encoded in a PCA layer and the loss imposed on the finallocation of the points. The end-to-end training makes it possible tostart from a random configuration of network parameters and find theoptimal set of filters and biases according to the estimation task andtraining data. This method may be applied to, for example, the landmarklocalization in 2D echocardiography images acquired from the parasternallong axis view and to the left ventricle segmentation of the heart inscans acquired from the apical four chamber view.

Incorporating statistical prior knowledge obtained through PCA into aconvolutional neural network may advantageously overcome the limitationsof previous deep learning approaches which lack strong shape priors andthe limitations of active shape models which lack advanced patternrecognition capabilities. This approach may be fully automatic andtherefore differs from most previous methods based on ASM which requiredhuman interaction. The neural network outputs the prediction in a singlestep without requiring any optimization loop.

In some embodiments, a training set containing N images and theassociated ground truth annotations consisting of coordinates referringto P key-points which describe the position of landmarks may beemployed. The training set may be used to first obtain the principalmodes of variation of the coordinates in Y and then train aconvolutional neural network that leverages it. The information used toformulate our predictions is obtained after multiple convolution andpooling operations and therefore fine-grained, high-resolution detailsmight be lost across the layers. For this reason, a mechanism may beemployed that focuses the attention of the network on full-resolutiondetails by cropping portions of the image in order to refine thepredictions. The architecture may be trained end-to-end, and all theparameters of the network may be updated at every iteration.

Much of the variability of naturally occurring structures, such asorgans and anatomical details of the body, is not arbitrary. By simpleobservation of a dataset of shapes representative of a population, forexample, one can notice the presence of symmetries and correlationsbetween different shape parts. In the same way, it is often possible toobserve correlations in the position of different landmarks of the bodysince they are tightly entangled with each other. PCA can be used todiscover the principal modes of variation of the dataset at hand. Whenshapes are described as aligned point sets across the entire dataset,PCA reveals what correlations exist between different points and definesa new coordinates frame where the principal modes of variationcorrespond to the axes. Having a matrix Y containing the dataset, whereeach observation y_(i) constitutes one of its columns, its principalcomponents may be obtained by first de-meaning Y through equation (3):

$\begin{matrix}{{\overset{\sim}{Y} = {Y - \mu}},{{{with}\mspace{14mu}\mu} = {\frac{1}{N}{\sum\limits_{i}\; y_{i}}}}} & (3)\end{matrix}$and then by computing the eigenvectors of the covariance matrix {tildeover (Y)}{tilde over (Y)}^(T). This corresponds to U in equation (4):

  (4)Which is obtained via singular value decomposition (SVD). The matrix isdiagonal and contains the eigenvalues of the covariance matrix andrepresent the variance associated with each principle component in theeigenbase.

Any example in the dataset can be synthesized as a linear combination ofthe principle components as shown in Equation (5):y _(i) =Uw+μ  (5)Each coefficient of the linear combination governs not only the positionof one, but multiple correlated points that may describe the shape athand. Imposing constraints on the coefficients weighting the effect ofeach principal component, or reducing their number until the correctbalance between percent-age of retained variance and number of principalcomponents is reached, it is possible to synthesize shapes that respectthe concept of “legal shape” introduced before.

The convolutional neural network may not be trained to performregression on the weights w in Equation 5. Instead, an end-to-endarchitecture may be used where the network directly uses the PCAeigenbase to make predictions from an image in the form of key-pointslocations. This has direct consequences on the training process. Thenetwork learns, by minimizing the loss, to steer the coefficients whilebeing “aware” of their effect on the results. Each of the weightscontrols the location of multiple correlated key-points simultaneously.Since the predictions are obtained as a linear combination of theprincipal components, they obey the concept of “legal shape” andtherefore are more robust to missing data, noise, and artifacts.

The network may comprises two branches. The first branch employsconvolutional, pooling, and dense layers, and produces a coarse estimateof the key-point locations via PCA. The second branch operates on fullresolution patches cropped from the input image around the coarsekey-point locations. The output of the second network refines thepredictions made by the first by using more fine-grained visualinformation. Both the branches are trained simultaneously and are fullydifferentiable. The convolutions are all applied without padding andthey use kernels of size 3×3 in the first convolutional neural networkbranch and 5×5 in the second, shallower, branch. The nonlinearities usedthroughout the network are rectified linear functions. All the inputs ofthe PCA layer, are not processed through nonlinearities.

The PCA layer implements a slightly modified of the synthesis equationin 5. In addition to the weights w, which are supplied by a dense layerof the network, a global shift s that is applied to all the predictedpoints is also supplied. Through the bi-dimensional vector s,translations of the anatomy of interest are able to be handled. With aslight abuse of notation, Equation 5 may be re-written as shown inEquation (6):y _(i) =Uw+μ+s.  (6)

The layer performing cropping follows an implementation inspired tospatial transformers which ensures differentiability. A regular samplingpattern is translated to the coarse key-point locations and theintensity values of the surrounding area are sampled using bilinearinterpolation. Having P key-points, P patches may be obtained for eachof the K images in the mini-batch. The resulting KP patches are thenprocessed through a 3-layers deep convolutional neural network using 8filters applied without padding, which reduces their size by a total of12 pixels. After the convolution layers, the patches are again arrangedinto a batch of K elements having P×8 channels, and further processedthrough three dense layers, which ultimately compute w_(A) having thesame dimensionality of w. The refined weights w_(F) which are employedin the PCA layer to obtain a more accurate key-point prediction, areobtained as w_(F)=w_(A)+w.

This approach has been tested on two different ultrasound datasetdepicting the human heart with the aim to solve two different tasks withgood results. The first task is segmentation of the left ventricle (LV)of the heart form scans acquired from the apical view, while the secondtask is a landmark localization problem where the aim is to localize 14points of interest in images acquired from the parasternal long axisview. In the first case, the model leverages prior statistical knowledgerelative to the shape of the structures of interest, while in the secondcase the model captures the spatiotemporal relationships betweenlandmarks across cardiac cycles of different patients. For thesegmentation task a total of 1100 annotated images, 953 for training and147 for testing, were employed.

Techniques for Landmark Localization Using Convolutional Neural Networks

The inventors have appreciated that accurate landmark localization inultrasound video sequences is challenging due to noise, shadows,anatomical differences, and scan plane variation. Accordingly, theinventors have conceived and developed a fully convolutional neuralnetwork trained to regress the landmark locations that may address suchchallenges. In this convolutional neural network, a series ofconvolution and pooling layers is followed by a collection of upsamplingand convolution layers with feature forwarding from the earlier layers.The final location estimates are produced by computing a center of massof the regression maps in the last layer. In addition, uncertainty ofthe estimates are computed as the standard deviations of thepredictions. The temporal consistency of the estimates is achieved by aLong Short-Term memory cells which processes several previous frames inorder to refine the estimate in the current frame. The results onautomatic measurement of left ventricle in parasternal long axis viewsand subsequent ejection fraction computation show accuracy on par withinter-user variability.

Regression modeling is an approach for describing relationship betweenan independent variable and one or more dependent variables. In machinelearning, this relationship is described by a function whose parametersare learned from training examples. In deep learning models, thisfunction is a composition of logistic (sigmoid), hyperbolic tangent, ormore recently rectified linear functions at each layer of the network.In many applications, the function learns a mapping between input imagepatches and a continuous prediction variable.

Regression modeling has been used to detect organ or landmark locationsin images, visually track objects and features, and estimate body poses.The deep learning approaches have outperformed previous techniquesespecially when a large annotated training data set is available. Theproposed architectures used cascade of regressors, refinementlocalization stages, and combining cues from multiple landmarks tolocalize landmarks. In medical images, the requirements on accuratelocalization are high since the landmarks are used as measurement pointsto help in diagnosis. When tracking the measurements in video sequences,the points must be accurately detected in each frame while ensuringtemporal consistency of the detections.

A fully convolutional network architecture for accurate localization ofanatomical landmark points in video sequences has been devised. Theadvantage of the fully convolutional network is that the responses frommultiple windows covering the input image can be computed in a singlestep. The network is trained end-to-end and outputs the locations of thelandmarks. The aggregation of the regressed locations at the lastconvolution layer is ensured by a new center-of-mass layer whichcomputes mean position of the predictions. The layer makes it possibleto use new regularization technique based on variance of the predictedcandidates and to define new loss based on relative locations oflandmarks. The evaluation is fast to process each frame of a videosequence at near frame rate speeds. The temporal consistency of themeasurements is improved by Convolutional Long Short-term Memory (CLSTM)cells which process the feature maps from several previous frames andproduce updated features for the current frame in order to refine theestimate.

Denote an input image of width w and height h as I (independentvariable) and the position of k landmarks stacked columnwise into p(dependent variable). The goal of the regression is to learn a functionƒ(I; θ)=p parametrized by θ. ƒ may be approximated by a convolutionalneural network and train the parameters params using a database ofimages and their corresponding annotations. Typically, a Euclidean lossis employed to train ƒ using each annotated image.

Previously, regression estimates were obtained directly from the lastlayer of the network, which was fully connected to previous layer. Thisis a highly non-linear mapping, where the estimate is computed from thefully connected layers after convolutional blocks. Instead of fullyconnected network, we propose to regress landmark locations using afully convolutional architecture (FCNN). Their advantage is that theestimates can be computed in a single evaluation step. In the proposedarchitecture, landmark coordinate estimates may be obtained at eachimage location.

The aggregated landmark coordinate estimates are computed in a newcenter of mass layer from input at each predicting location

$\begin{matrix}{\hat{p} = {\frac{1}{w \times h}{\sum\limits_{i = 1}^{h}\;{\sum\limits_{j = 1}^{w}\; l_{ij}}}}} & (7)\end{matrix}$

Recurrent neural networks (RNN) can learn sequential contextdependencies by accepting input x_(t) and updating a hidden vector h_(t)at every time step t. The RNN network can be composed of Long-short TermMemory (LSTM) units, each controlled by a gating mechanism with threetypes of updates, i_(t), ƒ_(t), o_(t) that range between 0 and 1. Thevalue i_(t) controls the update of each memory cell, ƒ_(t) controls theforgetting of each memory cell, and o_(t) controls the influence of thememory state on the hidden vector. In Convolutional LSTMs (CLSTMs), theinput weights and hidden vector weights are convolved instead ofmultiplied to model spatial constraints. The function introduces anon-linearity, which may be chosen to be tanh. Denoting theconvolutional operator as * for equations 8-10, the values at the gatesare computed as follows:forgetgate: ƒ_(t)=sign(W _(ƒ)*[h _(t-1) ,x _(t)]+b _(ƒ))  (8)inputgate: i _(t)=sign(W _(i)*[h _(t-1) ,x _(t)]+b _(i))  (9)outputgate: o _(t)=sign(W _(o)*[h _(t-1) ,x _(t)]+b _(o))  (10)

The parameters of the weights W and biases b are learned from trainingsequences. In addition to the gate values, each CLSTM unit computesstate candidate values:g _(t)=tan h(W _(g)*[h _(t-1) ,x _(t)]+b _(g))  (11)where g_(t) ranges between −1 and 1 and influences memory contents. Thememory cell is updated byc _(t)=ƒ_(t) ⊙c _(t-1) +i _(t) ⊙g _(t)  (12)which additively modifies each memory cell. The update process resultsin the gradients being distributed during backpropagation. The symboldenotes ⊙ the Hadamard product. Finally, the hidden state is updated as:h _(t) =o _(t)⊙ tan h(c _(t))  (13)

In sequential processing of image sequences, the inputs into the LSTMconsist of the feature maps computed from a convolutional neuralnetwork. In this work, two architectures are proposed to compute thefeature maps. The first architecture is a neural network withconvolution and pooling layers. After sequential processing the featuremaps in CLSTM, the output is fed into fully connected layers to computethe landmark location estimate. In the second architecture, the CLSTMinputs is the final layer of a convolutional path of the fullyconvolutional architecture (FCN). The landmark location estimates arecomputed from the CLSTM output processed through the transposedconvolutional part of the FCN network.

Example Ultrasound Systems

FIG. 15A is a schematic block diagram illustrating aspects of an exampleultrasound system 1500A upon which various aspects of the technologydescribed herein may be practiced. For example, one or more componentsof the ultrasound system 1500A may perform any of the processesdescribed herein. As shown, the ultrasound system 1500A comprisesprocessing circuitry 1501, input/output devices 1503, ultrasoundcircuitry 1505, and memory circuitry 1507.

The ultrasound circuitry 1505 may be configured to generate ultrasounddata that may be employed to generate an ultrasound image. Theultrasound circuitry 1505 may comprise one or more ultrasonictransducers monolithically integrated onto a single semiconductor die.The ultrasonic transducers may include, for example, one or morecapacitive micromachined ultrasonic transducers (CMUTs), one or moreCMOS ultrasonic transducers (CUTs), one or more piezoelectricmicromachined ultrasonic transducers (PMUTs), and/or one or more othersuitable ultrasonic transducer cells. In some embodiments, theultrasonic transducers may be formed the same chip as other electroniccomponents in the ultrasound circuitry 1505 (e.g., transmit circuitry,receive circuitry, control circuitry, power management circuitry, andprocessing circuitry) to form a monolithic ultrasound device.

The processing circuitry 1501 may be configured to perform any of thefunctionality described herein. The processing circuitry 1501 maycomprise one or more processors (e.g., computer hardware processors). Toperform one or more functions, the processing circuitry 1501 may executeone or more processor-executable instructions stored in the memorycircuitry 1507. The memory circuitry 1507 may be used for storingprograms and data during operation of the ultrasound system 1500B. Thememory circuitry 1507 may comprise one or more storage devices such asnon-transitory computer-readable storage media. The processing circuitry1501 may control writing data to and reading data from the memorycircuitry 1507 in any suitable manner.

In some embodiments, the processing circuitry 1501 may comprisespecially-programmed and/or special-purpose hardware such as anapplication-specific integrated circuit (ASIC). For example, theprocessing circuitry 1501 may comprise one or more tensor processingunits (TPUs). TPUs may be ASICs specifically designed for machinelearning (e.g., deep learning). The TPUs may be employed to, forexample, accelerate the inference phase of a neural network.

The input/output (I/O) devices 1503 may be configured to facilitatecommunication with other systems and/or an operator. Example I/O devicesthat may facilitate communication with an operator include: a keyboard,a mouse, a trackball, a microphone, a touch screen, a printing device, adisplay screen, a speaker, and a vibration device. Example I/O devicesthat may facilitate communication with other systems include wiredand/or wireless communication circuitry such as BLUETOOTH, ZIGBEE, WiFi,and/or USB communication circuitry.

It should be appreciated that the ultrasound system 1500A may beimplemented using any number of devices. For example, the components ofthe ultrasound system 1500A may be integrated into a single device. Inanother example, the ultrasound circuitry 1505 may be integrated into anultrasound device that is communicatively coupled with a computingdevice that comprises the processing circuitry 1501, the input/outputdevices 1503, and the memory circuitry 1507.

FIG. 15B is a schematic block diagram illustrating aspects of anotherexample ultrasound system 1500B upon which various aspects of thetechnology described herein may be practiced. For example, one or morecomponents of the ultrasound system 1500B may perform any of theprocesses described herein. As shown, the ultrasound system 1500Bcomprises an ultrasound device 1514 in wired and/or wirelesscommunication with a computing device 1502. The computing device 1502comprises an audio output device 1504, an imaging device 1506, a displayscreen 1508, a processor 1510, a memory 1512, and a vibration device1509. The computing device 1502 may communicate with one or moreexternal devices over a network 1516. For example, the computing device1502 may communicate with one or more workstations 1520, servers 1518,and/or databases 1522.

The ultrasound device 1514 may be configured to generate ultrasound datathat may be employed to generate an ultrasound image. The ultrasounddevice 1514 may be constructed in any of a variety of ways. In someembodiments, the ultrasound device 1514 includes a transmitter thattransmits a signal to a transmit beamformer which in turn drivestransducer elements within a transducer array to emit pulsed ultrasonicsignals into a structure, such as a patient. The pulsed ultrasonicsignals may be back-scattered from structures in the body, such as bloodcells or muscular tissue, to produce echoes that return to thetransducer elements. These echoes may then be converted into electricalsignals, or ultrasound data, by the transducer elements and theelectrical signals are received by a receiver. The electrical signalsrepresenting the received echoes are sent to a receive beamformer thatoutputs ultrasound data.

The computing device 1502 may be configured to process the ultrasounddata from the ultrasound device 1514 to generate ultrasound images fordisplay on the display screen 1508. The processing may be performed by,for example, the processor 1510. The processor 1510 may also be adaptedto control the acquisition of ultrasound data with the ultrasound device1514. The ultrasound data may be processed in real-time during ascanning session as the echo signals are received. In some embodiments,the displayed ultrasound image may be updated a rate of at least 5 Hz,at least 10 Hz, at least 20 Hz, at a rate between 5 and 60 Hz, at a rateof more than 20 Hz. For example, ultrasound data may be acquired even asimages are being generated based on previously acquired data and while alive ultrasound image is being displayed. As additional ultrasound datais acquired, additional frames or images generated from more-recentlyacquired ultrasound data are sequentially displayed. Additionally, oralternatively, the ultrasound data may be stored temporarily in a bufferduring a scanning session and processed in less than real-time.

Additionally (or alternatively), the computing device 1502 may beconfigured to perform any of the processes described herein (e.g., usingthe processor 1510) and/or display any of the user interfaces describedherein (e.g., using the display screen 1508). For example, the computingdevice 1502 may be configured to provide instructions to an operator ofthe ultrasound device 1514 to assist the operator select a targetanatomical view of a subject to image and to guide the operator capturean ultrasound image containing the target anatomical view. As shown, thecomputing device 1502 may comprise one or more elements that may be usedduring the performance of such processes. For example, the computingdevice 1502 may comprise one or more processors 1510 (e.g., computerhardware processors) and one or more articles of manufacture thatcomprise non-transitory computer-readable storage media such as thememory 1512. The processor 1510 may control writing data to and readingdata from the memory 1512 in any suitable manner. To perform any of thefunctionality described herein, the processor 1510 may execute one ormore processor-executable instructions stored in one or morenon-transitory computer-readable storage media (e.g., the memory 1512),which may serve as non-transitory computer-readable storage mediastoring processor-executable instructions for execution by the processor1510.

In some embodiments, the computing device 1502 may comprise one or moreinput and/or output devices such as the audio output device 1504, theimaging device 1506, the display screen 1508, and the vibration device1509. The audio output device 1504 may be a device that is configured toemit audible sound such as a speaker. The imaging device 1506 may beconfigured to detect light (e.g., visible light) to form an image suchas a camera. The display screen 1508 may be configured to display imagesand/or videos such as a liquid crystal display (LCD), a plasma display,and/or an organic light emitting diode (OLED) display. The vibrationdevice 1509 may be configured to vibrate one or more components of thecomputing device 1502 to provide tactile feedback. These input and/oroutput devices may be communicatively coupled to the processor 1510and/or under the control of the processor 1510. The processor 1510 maycontrol these devices in accordance with a process being executed by theprocess 1510 (such as any of the processes shown in FIGS. 9-13). Forexample, the processor 1510 may control the display screen 1508 todisplay any of the above described user interfaces, instructions, and/orultrasound images. Similarly, the processor 1510 may control the audiooutput device 1504 to issue audible instructions and/or control thevibration device 1509 to change an intensity of tactile feedback (e.g.,vibration) to issue tactile instructions. Additionally (oralternatively), the processor 1510 may control the imaging device 1506to capture non-acoustic images of the ultrasound device 1514 being usedon a subject to provide an operator of the ultrasound device 1514 anaugmented reality interface (e.g., as shown in FIGS. 5B and 6).

It should be appreciated that the computing device 1502 may beimplemented in any of a variety of ways. For example, the computingdevice 1502 may be implemented as a handheld device such as a mobilesmartphone or a tablet. Thereby, an operator of the ultrasound device1514 may be able to operate the ultrasound device 1514 with one hand andhold the computing device 1502 with another hand. In other examples, thecomputing device 1502 may be implemented as a portable device that isnot a handheld device such as a laptop. In yet other examples, thecomputing device 1502 may be implemented as a stationary device such asa desktop computer.

In some embodiments, the computing device 1502 may communicate with oneor more external devices via the network 1516. The computing device 1502may be connected to the network 1516 over a wired connection (e.g., viaan Ethernet cable) and/or a wireless connection (e.g., over a WiFinetwork). As shown in FIG. 15B, these external devices may includeservers 1518, workstations 1520, and/or databases 1522. The computingdevice 1502 may communicate with these devices to, for example, off-loadcomputationally intensive tasks. For example, the computing device 1502may send an ultrasound image over the network 1516 to the server 1518for analysis (e.g., to identify an anatomical feature in the ultrasoundimage and/or identify an instruction to provide the operator) andreceive the results of the analysis from the server 1518. Additionally(or alternatively), the computing device 1502 may communicate with thesedevices to access information that is not available locally and/orupdate a central information repository. For example, the computingdevice 1502 may access the medical records of a subject being imagedwith the ultrasound device 1514 from a file stored in the database 1522.In this example, the computing device 1502 may also provide one or morecaptured ultrasound images of the subject to the database 1522 to add tothe medical record of the subject.

The terms “program,” “application,” or “software” are used herein in ageneric sense to refer to any type of computer code or set ofprocessor-executable instructions that may be employed to program acomputer or other processor to implement various aspects of embodimentsas discussed above. Additionally, according to one aspect, one or morecomputer programs that when executed perform methods of the disclosureprovided herein need not reside on a single computer or processor, butmay be distributed in a modular fashion among different computers orprocessors to implement various aspects of the disclosure providedherein.

Processor-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically, the functionality of the program modulesmay be combined or distributed.

Also, data structures may be stored in one or more non-transitorycomputer-readable storage media in any suitable form. For simplicity ofillustration, data structures may be shown to have fields that arerelated through location in the data structure. Such relationships maylikewise be achieved by assigning storage for the fields with locationsin a non-transitory computer-readable medium that convey relationshipbetween the fields. However, any suitable mechanism may be used toestablish relationships among information in fields of a data structure,including through the use of pointers, tags or other mechanisms thatestablish relationships among data elements.

Example Ultrasound Devices

FIG. 16 shows an illustrative example of a monolithic ultrasound device1600 that may be employed as any of the ultrasound devices describedabove such as ultrasound devices 102, 502, 602, and 1514 or any of theultrasound circuitry described herein such as ultrasound circuitry 1505.As shown, the ultrasound device 1600 may include one or more transducerarrangements (e.g., arrays) 1602, transmit (TX) circuitry 1604, receive(RX) circuitry 1606, a timing and control circuit 1608, a signalconditioning/processing circuit 1610, a power management circuit 1618,and/or a high-intensity focused ultrasound (HIFU) controller 1620. Inthe embodiment shown, all of the illustrated elements are formed on asingle semiconductor die 1612. It should be appreciated, however, thatin alternative embodiments one or more of the illustrated elements maybe instead located off-chip. In addition, although the illustratedexample shows both TX circuitry 1604 and RX circuitry 1606, inalternative embodiments only TX circuitry or only RX circuitry may beemployed. For example, such embodiments may be employed in acircumstance where one or more transmission-only devices 1600 are usedto transmit acoustic signals and one or more reception-only devices 1600are used to receive acoustic signals that have been transmitted throughor reflected off of a subject being ultrasonically imaged.

It should be appreciated that communication between one or more of theillustrated components may be performed in any of numerous ways. In someembodiments, for example, one or more high-speed busses (not shown),such as that employed by a unified Northbridge, may be used to allowhigh-speed intra-chip communication or communication with one or moreoff-chip components.

The one or more transducer arrays 1602 may take on any of numerousforms, and aspects of the present technology do not necessarily requirethe use of any particular type or arrangement of transducer cells ortransducer elements. Indeed, although the term “array” is used in thisdescription, it should be appreciated that in some embodiments thetransducer elements may not be organized in an array and may instead bearranged in some non-array fashion. In various embodiments, each of thetransducer elements in the array 1602 may, for example, include one ormore capacitive micromachined ultrasonic transducers (CMUTs), one ormore CMOS ultrasonic transducers (CUTs), one or more piezoelectricmicromachined ultrasonic transducers (PMUTs), and/or one or more othersuitable ultrasonic transducer cells. In some embodiments, thetransducer elements of the transducer array 102 may be formed on thesame chip as the electronics of the TX circuitry 1604 and/or RXcircuitry 1606. The transducer elements 1602, TX circuitry 1604, and RXcircuitry 1606 may, in some embodiments, be integrated in a singleultrasound device. In some embodiments, the single ultrasound device maybe a handheld device. In other embodiments, the single ultrasound devicemay be embodied in a patch that may be coupled to a patient. The patchmay be configured to transmit, wirelessly, data collected by the patchto one or more external devices for further processing.

A CUT may, for example, include a cavity formed in a CMOS wafer, with amembrane overlying the cavity, and in some embodiments sealing thecavity. Electrodes may be provided to create a transducer cell from thecovered cavity structure. The CMOS wafer may include integratedcircuitry to which the transducer cell may be connected. The transducercell and CMOS wafer may be monolithically integrated, thus forming anintegrated ultrasonic transducer cell and integrated circuit on a singlesubstrate (the CMOS wafer).

The TX circuitry 1604 (if included) may, for example, generate pulsesthat drive the individual elements of, or one or more groups of elementswithin, the transducer array(s) 1602 so as to generate acoustic signalsto be used for imaging. The RX circuitry 1606, on the other hand, mayreceive and process electronic signals generated by the individualelements of the transducer array(s) 102 when acoustic signals impingeupon such elements.

In some embodiments, the timing and control circuit 1608 may, forexample, be responsible for generating all timing and control signalsthat are used to synchronize and coordinate the operation of the otherelements in the device 1600. In the example shown, the timing andcontrol circuit 1608 is driven by a single clock signal CLK supplied toan input port 1616. The clock signal CLK may, for example, be ahigh-frequency clock used to drive one or more of the on-chip circuitcomponents. In some embodiments, the clock signal CLK may, for example,be a 1.5625 GHz or 2.5 GHz clock used to drive a high-speed serialoutput device (not shown in FIG. 16) in the signalconditioning/processing circuit 1610, or a 20 Mhz or 40 MHz clock usedto drive other digital components on the semiconductor die 1612, and thetiming and control circuit 1608 may divide or multiply the clock CLK, asnecessary, to drive other components on the die 1612. In otherembodiments, two or more clocks of different frequencies (such as thosereferenced above) may be separately supplied to the timing and controlcircuit 1608 from an off-chip source.

The power management circuit 1618 may, for example, be responsible forconverting one or more input voltages VIN from an off-chip source intovoltages needed to carry out operation of the chip, and for otherwisemanaging power consumption within the device 1600. In some embodiments,for example, a single voltage (e.g., 12V, 80V, 100V, 120V, etc.) may besupplied to the chip and the power management circuit 1618 may step thatvoltage up or down, as necessary, using a charge pump circuit or viasome other DC-to-DC voltage conversion mechanism. In other embodiments,multiple different voltages may be supplied separately to the powermanagement circuit 1618 for processing and/or distribution to the otheron-chip components.

As shown in FIG. 16, in some embodiments, a HIFU controller 1620 may beintegrated on the semiconductor die 1612 so as to enable the generationof HIFU signals via one or more elements of the transducer array(s)1602. In other embodiments, a HIFU controller for driving the transducerarray(s) 1602 may be located off-chip, or even within a device separatefrom the device 1600. That is, aspects of the present disclosure relateto provision of ultrasound-on-a-chip HIFU systems, with and withoutultrasound imaging capability. It should be appreciated, however, thatsome embodiments may not have any HIFU capabilities and thus may notinclude a HIFU controller 1620.

Moreover, it should be appreciated that the HIFU controller 1620 may notrepresent distinct circuitry in those embodiments providing HIFUfunctionality. For example, in some embodiments, the remaining circuitryof FIG. 16 (other than the HIFU controller 1620) may be suitable toprovide ultrasound imaging functionality and/or HIFU, i.e., in someembodiments the same shared circuitry may be operated as an imagingsystem and/or for HIFU. Whether or not imaging or HIFU functionality isexhibited may depend on the power provided to the system. HIFU typicallyoperates at higher powers than ultrasound imaging. Thus, providing thesystem a first power level (or voltage level) appropriate for imagingapplications may cause the system to operate as an imaging system,whereas providing a higher power level (or voltage level) may cause thesystem to operate for HIFU. Such power management may be provided byoff-chip control circuitry in some embodiments.

In addition to using different power levels, imaging and HIFUapplications may utilize different waveforms. Thus, waveform generationcircuitry may be used to provide suitable waveforms for operating thesystem as either an imaging system or a HIFU system.

In some embodiments, the system may operate as both an imaging systemand a HIFU system (e.g., capable of providing image-guided HIFU). Insome such embodiments, the same on-chip circuitry may be utilized toprovide both functions, with suitable timing sequences used to controlthe operation between the two modalities.

In the example shown, one or more output ports 1614 may output ahigh-speed serial data stream generated by one or more components of thesignal conditioning/processing circuit 1610. Such data streams may, forexample, be generated by one or more USB 3.0 modules, and/or one or more10 GB, 40 GB, or 100 GB Ethernet modules, integrated on thesemiconductor die 1612. In some embodiments, the signal stream producedon output port 1614 can be fed to a computer, tablet, or smartphone forthe generation and/or display of 2-dimensional, 3-dimensional, and/ortomographic images. In embodiments in which image formation capabilitiesare incorporated in the signal conditioning/processing circuit 1610,even relatively low-power devices, such as smartphones or tablets whichhave only a limited amount of processing power and memory available forapplication execution, can display images using only a serial datastream from the output port 1614. As noted above, the use of on-chipanalog-to-digital conversion and a high-speed serial data link tooffload a digital data stream is one of the features that helpsfacilitate an “ultrasound on a chip” solution according to someembodiments of the technology described herein.

Devices 1600 such as that shown in FIG. 16 may be used in any of anumber of imaging and/or treatment (e.g., HIFU) applications, and theparticular examples discussed herein should not be viewed as limiting.In one illustrative implementation, for example, an imaging deviceincluding an N×M planar or substantially planar array of CMUT elementsmay itself be used to acquire an ultrasonic image of a subject, e.g., aperson's abdomen, by energizing some or all of the elements in thearray(s) 1602 (either together or individually) during one or moretransmit phases, and receiving and processing signals generated by someor all of the elements in the array(s) 1602 during one or more receivephases, such that during each receive phase the CMUT elements senseacoustic signals reflected by the subject. In other implementations,some of the elements in the array(s) 1602 may be used only to transmitacoustic signals and other elements in the same array(s) 1602 may besimultaneously used only to receive acoustic signals. Moreover, in someimplementations, a single imaging device may include a P×Q array ofindividual devices, or a P×Q array of individual N×M planar arrays ofCMUT elements, which components can be operated in parallel,sequentially, or according to some other timing scheme so as to allowdata to be accumulated from a larger number of CMUT elements than can beembodied in a single device 1600 or on a single die 1612.

In yet other implementations, a pair of imaging devices can bepositioned so as to straddle a subject, such that one or more CMUTelements in the device(s) 1600 of the imaging device on one side of thesubject can sense acoustic signals generated by one or more CMUTelements in the device(s) 1600 of the imaging device on the other sideof the subject, to the extent that such pulses were not substantiallyattenuated by the subject. Moreover, in some implementations, the samedevice 1600 can be used to measure both the scattering of acousticsignals from one or more of its own CMUT elements as well as thetransmission of acoustic signals from one or more of the CMUT elementsdisposed in an imaging device on the opposite side of the subject.

FIG. 17 is a block diagram illustrating how, in some embodiments, the TXcircuitry 1604 and the RX circuitry 1606 for a given transducer element1702 may be used either to energize the transducer element 1702 to emitan ultrasonic pulse, or to receive and process a signal from thetransducer element 1702 representing an ultrasonic pulse sensed by it.In some implementations, the TX circuitry 1604 may be used during a“transmission” phase, and the RX circuitry may be used during a“reception” phase that is non-overlapping with the transmission phase.In other implementations, one of the TX circuitry 1604 and the RXcircuitry 1606 may simply not be used in a given device 1600, such aswhen a pair of ultrasound units is used for only transmissive imaging.As noted above, in some embodiments, an ultrasound device 1600 mayalternatively employ only TX circuitry 1604 or only RX circuitry 1606,and aspects of the present technology do not necessarily require thepresence of both such types of circuitry. In various embodiments, TXcircuitry 1604 and/or RX circuitry 1606 may include a TX circuit and/oran RX circuit associated with a single transducer cell (e.g., a CUT orCMUT), a group of two or more transducer cells within a singletransducer element 1702, a single transducer element 1702 comprising agroup of transducer cells, a group of two or more transducer elements1702 within an array 1602, or an entire array 1602 of transducerelements 1702.

In the example shown in FIG. 17, the TX circuitry 1604/RX circuitry 1606includes a separate TX circuit and a separate RX circuit for eachtransducer element 1702 in the array(s) 1602, but there is only oneinstance of each of the timing & control circuit 1608 and the signalconditioning/processing circuit 1610. Accordingly, in such animplementation, the timing & control circuit 1608 may be responsible forsynchronizing and coordinating the operation of all of the TX circuitry1604/RX circuitry 1606 combinations on the die 1612, and the signalconditioning/processing circuit 1610 may be responsible for handlinginputs from all of the RX circuitry 1606 on the die 1612. In otherembodiments, timing and control circuit 1608 may be replicated for eachtransducer element 1702 or for a group of transducer elements 1702.

As shown in FIG. 17, in addition to generating and/or distributing clocksignals to drive the various digital components in the device 1600, thetiming & control circuit 1608 may output either an “TX enable” signal toenable the operation of each TX circuit of the TX circuitry 1604, or an“RX enable” signal to enable operation of each RX circuit of the RXcircuitry 1606. In the example shown, a switch 1716 in the RX circuitry1606 may always be opened before the TX circuitry 1604 is enabled, so asto prevent an output of the TX circuitry 1604 from driving the RXcircuitry 1606. The switch 1716 may be closed when operation of the RXcircuitry 1606 is enabled, so as to allow the RX circuitry 1606 toreceive and process a signal generated by the transducer element 1702.

As shown, the TX circuitry 1604 for a respective transducer element 1702may include both a waveform generator 1714 and a pulser 1712. Thewaveform generator 1714 may, for example, be responsible for generatinga waveform that is to be applied to the pulser 1712, so as to cause thepulser 1712 to output a driving signal to the transducer element 1702corresponding to the generated waveform.

In the example shown in FIG. 17, the RX circuitry 1606 for a respectivetransducer element 1702 includes an analog processing block 1718, ananalog-to-digital converter (ADC) 1720, and a digital processing block1722. The ADC 1720 may, for example, comprise a 10-bit or 12-bit, 20Msps, 25 Msps, 40 Msps, 50 Msps, or 80 Msps ADC.

After undergoing processing in the digital processing block 1722, theoutputs of all of the RX circuits on the semiconductor die 1612 (thenumber of which, in this example, is equal to the number of transducerelements 1702 on the chip) are fed to a multiplexer (MUX) 1724 in thesignal conditioning/processing circuit 1610. In other embodiments, thenumber of transducer elements is larger than the number of RX circuits,and several transducer elements provide signals to a single RX circuit.The MUX 1724 multiplexes the digital data from the RX circuits, and theoutput of the MUX 1724 is fed to a multiplexed digital processing block1726 in the signal conditioning/processing circuit 1610, for finalprocessing before the data is output from the semiconductor die 1612,e.g., via one or more high-speed serial output ports 1614. The MUX 1724is optional, and in some embodiments parallel signal processing isperformed. A high-speed serial data port may be provided at anyinterface between or within blocks, any interface between chips and/orany interface to a host. Various components in the analog processingblock 1718 and/or the digital processing block 1722 may reduce theamount of data that needs to be output from the semiconductor die 1612via a high-speed serial data link or otherwise. In some embodiments, forexample, one or more components in the analog processing block 1718and/or the digital processing block 1722 may thus serve to allow the RXcircuitry 1606 to receive transmitted and/or scattered ultrasoundpressure waves with an improved signal-to-noise ratio (SNR) and in amanner compatible with a diversity of waveforms. The inclusion of suchelements may thus further facilitate and/or enhance the disclosed“ultrasound-on-a-chip” solution in some embodiments.

The ultrasound devices described herein may be implemented in any of avariety of physical configurations including as part of a handhelddevice (which may include a screen to display obtained images) or aspart of a patch configured to be affixed to the subject.

In some embodiments, an ultrasound device may be embodied in a handhelddevice 1802 illustrated in FIGS. 18A and 18B. Handheld device 1802 maybe held against (or near) a subject 1800 and used to image the subject.Handheld device 1802 may comprise an ultrasound device and display 1804,which in some embodiments, may be a touchscreen. Display 1804 may beconfigured to display images of the subject (e.g., ultrasound images)generated within handheld device 1802 using ultrasound data gathered bythe ultrasound device within device 1802.

In some embodiments, handheld device 1802 may be used in a manneranalogous to a stethoscope. A medical professional may place handhelddevice 1802 at various positions along a patient's body. The ultrasounddevice within handheld device 1802 may image the patient. The dataobtained by the ultrasound device may be processed and used to generateimage(s) of the patient, which image(s) may be displayed to the medicalprofessional via display 1804. As such, a medical professional couldcarry the handheld device 1802 (e.g., around their neck or in theirpocket) rather than carrying around multiple conventional devices, whichis burdensome and impractical.

In some embodiments, an ultrasound device may be embodied in a patchthat may be coupled to a patient. For example, FIGS. 18C and 18Dillustrate a patch 1810 coupled to patient 1812. The patch 1810 may beconfigured to transmit, wirelessly, data collected by the patch 1810 toone or more external devices for further processing. FIG. 18E shows anexploded view of patch 1810.

In some embodiments, an ultrasound device may be embodied in handhelddevice 1820 shown in FIG. 18F. Handheld device 1820 may be configured totransmit data collected by the device 1820 wirelessly to one or moreexternal device for further processing. In other embodiments, handhelddevice 1820 may be configured transmit data collected by the device 1820to one or more external devices using one or more wired connections, asaspects of the technology described herein are not limited in thisrespect.

Various aspects of the present disclosure may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and is therefore notlimited in its application to the details and arrangement of componentsset forth in the foregoing description or illustrated in the drawings.For example, aspects described in one embodiment may be combined in anymanner with aspects described in other embodiments.

Further, some actions are described as taken by a “operator” or“subject.” It should be appreciated that a “operator” or “subject” neednot be a single individual, and that in some embodiments, actionsattributable to an “operator” or “subject” may be performed by a team ofindividuals and/or an individual in combination with computer-assistedtools or other mechanisms. Further, it should be appreciated that, insome instances, a “subject” may be the same person as the “operator.”For example, an individual may be imaging themselves with an ultrasounddevice and, thereby, act as both the “subject” being imaged and the“operator” of the ultrasound device.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

The terms “approximately” and “about” may be used to mean within ±20% ofa target value in some embodiments, within ±10% of a target value insome embodiments, within ±5% of a target value in some embodiments, andyet within ±2% of a target value in some embodiments. The terms“approximately” and “about” may include the target value.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

Having described above several aspects of at least one embodiment, it isto be appreciated various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements are intended to be object of thisdisclosure. Accordingly, the foregoing description and drawings are byway of example only.

What is claimed is:
 1. A method for guiding an operator of an ultrasounddevice in positioning the ultrasound device, the method comprising:using a single mobile device, separate from the ultrasound device andcomprising at least one processor, a camera, and a display, to perform:automatically identifying a pose of the ultrasound device at least inpart by imaging the ultrasound device with the camera of the singlemobile device; generating, using the single mobile device, a compositeimage of the ultrasound device and an indication for how to move theultrasound device, the generating comprising overlaying the indicationon an image of the ultrasound device at a location in the imagedetermined using the identified pose of the ultrasound device; andpresenting the composite image to the operator of the ultrasound deviceusing the display of the single mobile device.
 2. The method of claim 1,wherein automatically identifying the pose of the ultrasound devicecomprises identifying a location, in the image, of a marker on theultrasound device.
 3. The method of claim 2, wherein automaticallyidentifying the pose of the ultrasound device comprises identifying aposition of the ultrasound device in the image using the identifiedlocation of the marker in the image.
 4. The method of claim 1, whereinthe ultrasound device comprises at least one sensor configured to detectmovement of the ultrasound device, and wherein automatically identifyingthe pose of the ultrasound device is performed using data collected bythe at least one sensor.
 5. The method of claim 4, wherein the at leastone sensor comprises a gyroscope.
 6. The method of claim 4, wherein theat least one sensor comprises an accelerometer.
 7. The method of claim1, further comprising: obtaining an ultrasound image captured by theultrasound device; and generating the indication using the ultrasoundimage.
 8. The method of claim 7, further comprising: presenting theultrasound image to the operator of the ultrasound device using thedisplay of the mobile device.
 9. A system for guiding an operator of anultrasound device in positioning the ultrasound device, the systemcomprising: a single mobile device, separate from the ultrasound deviceand comprising at least one processor, a camera, and a display, thesingle mobile device configured to: automatically identify a pose of theultrasound device at least in part by imaging the ultrasound device withthe camera of the single mobile device; generate, using the singlemobile device, a composite image of the ultrasound device and anindication for how to move the ultrasound device, the generatingcomprising overlaying the indication on an image of the ultrasounddevice at a location in the image determined using the identified poseof the ultrasound device; and present the composite image to theoperator of the ultrasound device using the display of the single mobiledevice.
 10. The system of claim 9, further comprising: the ultrasounddevice.
 11. The system of claim 10, wherein the ultrasound devicecomprises at least one sensor configured to detect movement of theultrasound device.
 12. The system of claim 9, wherein the single mobiledevice is configured to automatically identify the pose of theultrasound device at least in part by identifying a location, in theimage, of a marker on the ultrasound device.
 13. The system of claim 12,wherein the single mobile device is configured to automatically identifythe pose of the ultrasound device at least in part by identifying aposition of the ultrasound device in the image using the identifiedlocation of the marker in the image.
 14. The system of claim 11, whereinthe single mobile device is configured to automatically identify thepose of the ultrasound device at least in part by using data collectedby the at least one sensor.
 15. The system of claim 14, wherein the atleast one sensor comprises a gyroscope.
 16. The system of claim 14,wherein the at least one sensor comprises an accelerometer.
 17. At leastone non-transitory computer-readable storage medium storingprocessor-executable instructions that, when executed by a single mobiledevice, the single mobile device comprising at least one processor, acamera, and a display, cause the single mobile device to: automaticallyidentify a pose of an ultrasound device at least in part by imaging theultrasound device with the camera of the single mobile device; generate,using the single mobile device, a composite image of the ultrasounddevice and an indication for how to move the ultrasound device, thegenerating comprising overlaying the indication on an image of theultrasound device at a location in the image determined using theidentified pose of the ultrasound device; and present the compositeimage to an operator of the ultrasound device using the display of thesingle mobile device.
 18. The method of claim 1, wherein overlaying theindication on the image of the ultrasound device comprises overlaying anindication to move the ultrasound device in a specific direction. 19.The method of claim 18, wherein overlaying the indication to move theultrasound device in the specific direction comprises overlaying, on theimage of the ultrasound device, an arrow pointing in the direction. 20.The method of claim 1, wherein overlaying the indication on the image ofthe ultrasound device comprises overlaying an indication to rotate theultrasound device.
 21. The system of claim 9, wherein overlaying theindication on the image of the ultrasound device comprises overlaying anindication to move the ultrasound device in a specific direction. 22.The system of claim 9, wherein overlaying the indication on the image ofthe ultrasound device comprises overlaying an indication to rotate theultrasound device.
 23. The at least one non-transitory computer-readablestorage medium of claim 17, wherein overlaying the indication on theimage of the ultrasound device comprises overlaying an indication tomove the ultrasound device in a specific direction.
 24. The at least onenon-transitory computer-readable storage medium of claim 17, whereinoverlaying the indication on the image of the ultrasound devicecomprises overlaying an indication to rotate the ultrasound device. 25.The at least one non-transitory computer-readable storage medium ofclaim 17, wherein the single mobile device is configured toautomatically identify the pose of the ultrasound device at least inpart by identifying a location, in the image, of a marker on theultrasound device.
 26. The at least one non-transitory computer-readablestorage medium of claim 25, wherein the single mobile device isconfigured to automatically identify the pose of the ultrasound deviceat least in part by identifying a position of the ultrasound device inthe image using the identified location of the marker in the image. 27.The at least one non-transitory computer-readable storage medium ofclaim 17, wherein the ultrasound device comprises at least one sensorconfigured to detect movement of the ultrasound device, wherein thesingle mobile device is configured to automatically identify the pose ofthe ultrasound device at least in part by using data collected by the atleast one sensor.
 28. The at least one non-transitory computer-readablestorage medium of claim 27, wherein the at least one sensor comprises agyroscope.